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Executive Summary 


The Wireless Emergency Alerts (WEA) service, formerly known as the Commercial Mobile Alert 
Service (CMAS), enhances public safety by providing authorized emergency management agen¬ 
cies (EMAs) with the capability to issue alerts and warnings to mobile communication devices 
(e.g., cell phones) in a designated geographic area. WEA is a component of the Integrated Public 
Alert and Warning System (IPAWS) operated by the Federal Emergency Management Agency 
(FEMA) in cooperation with the Federal Communications Commission (FCC) and supported by 
the Department of Flomeland Security Science and Technology Directorate (DFIS S&T). Three 
categories of WEA messages may be sent: Presidential Alerts, for events of critical importance to 
the nation; Imminent Threat Alerts, for threats arising from weather and other hazards; and Amer¬ 
icas Missing: Broadcast Emergency Response Alerts, for missing children. 

Trust is a key factor in the effectiveness of the WEA service. Alert originators (AOs) at EMAs 
must trust WEA to deliver alerts to the public in an accurate and timely manner. Absent this trast, 
AOs will not use WEA. Members of the public must also trust the WEA service. They must un¬ 
derstand and believe the messages that they receive before they will act on them. Clearly, FEMA, 
the EMAs, and the AOs must all strive to maximize and maintain trust in the WEA service if it is 
to be an effective alerting tool. 

In 2012, DHS S&T tasked the Carnegie Mellon® Software Engineering Institute (SEI) with de¬ 
veloping a WEA trust model. The purpose of this model was to provide data for FEMA that 
would enable it to maximize the effectiveness of WEA and provide guidance for AOs that would 
support them in using WEA in a manner that maximized public safety. Our approach to this task 
was to 

1. build models that could predict the levels of AO trust and public trust in specific scenarios 

2. validate these models using data collected from AOs and the public 

3. execute simulations on these models for numerous scenarios to identify 

a. recommendations to AOs and/or FEMA for actions to take that increase tmst 

b. recommendations to AOs and/or FEMA for actions to avoid that decrease trust 

Results of this work consist of 

1. a detailed technical report (this report) describing the process employed in the development 
and validation of the trust models and the resulting structure and functionality of the models 

2. Trust Model Simulations for the Wireless Emergency Alerts (WEA) Service, a report detailing 
the scenarios and simulations executed on the trast models [Morrow 2013] 

3. Maximizing Trust in the Wireless Emergency Alerts (WEA) Service, a recommendations re¬ 
port analyzing the results of the simulations and identifying trust-enhancing practices to be 
employed and trust-degrading processes to be avoided by both AOs and FEMA [Woody 
2013] 

This project began with a review of the literature addressing issues of trast in public alerting. As a 
result of this review, we adjusted the project scope to focus primarily on “trast” factors considered 
within the influence or control of WEA system operators and AOs. We realized that most, if not 
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all, potential factors associated with trust in the WEA system would be subjective in nature. This 
realization suggested the use of Bayesian belief networks (BBNs) to model trust. It also suggested 
that we would need to perform extensive data collection through interviews and surveys of AOs 
and representatives of the public who would be receiving these alerts. To facilitate this data col¬ 
lection, we approached the CMU Office of Research Integrity and Compliance for guidance and 
approval with regard to the human-subject research aspects of the WEA trust model development. 
The literature review enabled the development of an interview script containing 25 interview 
questions, which were then administered to 17 emergency alert organizations. 


From these initial interviews, we identified 56 preliminary factors as potentially influential on the 
public’s and/or AOs’ trust in the WEA system. Using cause-effect mapping tables, we consoli¬ 
dated the tmst factor lists into a more manageable list of 27 trust factors for the public trast model 
and 35 trust factors for the AO trust model. Armed with the list of tmst factors for each model, we 
then developed surveys to canvas members of the public and AO communities for their responses 
to hypothetical alert message scenarios. Consequently, we developed three surveys of approxi¬ 
mately 19 questions each to administer to equal subsets of approximately 5,000 members of the 
public. We also developed three surveys of approximately 12 questions each to administer to 
equal subsets of approximately 560 AOs. After subjecting the data collected from the surveys to 
statistical analysis involving hypothesis testing and linear regression modeling, we concluded that 
only 7 of the relationships in the public tmst model were statistically significant at the 5% level 
while 29 of the relationships in the AO tmst model were statistically significant at the 5% level. 
That does not mean that the remaining relationships did not influence trust; it means that we need 
more survey data to achieve a larger sample than we were able to collect. Such a sample would 
reduce the measurement noise and enable us to determine the relationship. 


Once we had developed the models, we exercised each with a series of scenarios addressing all of 
the model inputs and outputs. We also presented these same scenarios to panels composed of rep¬ 
resentatives of the public and alerting subject-matter experts. For each BBN model, we related the 
probability predictions of tmst outcomes to the results from corresponding validation scenarios. 
The result was a collection of linear regression equations that capably “calibrate” the BBN predic¬ 
tions to final validation scenario results, as shown in Figure 1. 
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Figure 1: Trust Model Validation Process 

We expected such a transform to be necessary because the BBN models were populated based on 
1:1 factor assessments and ignored possible factor interactions. Hence, the regression equations 
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used to transform the BBN prediction probabilities account for both missing interaction effects 
and measurement error associated with the small sample of validation scenario evaluations. In the 
final analysis, the BBN trast models facilitated the operational analysis of additional and unprece¬ 
dented scenarios in support of a separate effort to develop guidance for AOs and WEA system 
operators concerned with trust in the WEA system. We also identified opportunities for future 
research that could leverage the operational nature of the BBN trust models in AgenaRisk, name¬ 
ly, the conduct of observational research into the trust factors, making use of the learning mecha¬ 
nisms possible within BBN models. 
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Abstract 


Trust is a key factor in the effectiveness of the Wireless Emergency Alerts (WEA) service. Alert 
originators (AOs) must trast WEA to deliver alerts to the public in an accurate and timely manner. 
Members of the public must also trust the WEA service before they will act on the alerts that they 
receive. This research aimed to develop a trust model to enable the Federal Emergency Manage¬ 
ment Agency (FEMA) to maximize the effectiveness of WEA and provide guidance for AOs that 
would support them in using WEA in a manner that maximizes public safety. The research meth¬ 
od included Bayesian belief networks to model trust in WEA because they enable reasoning about 
and modeling of uncertainty. The research approach was to build models that could predict the 
levels of AO tmst and public trust in specific scenarios, validate these models using data collected 
from AOs and the public, and execute simulations on these models for numerous scenarios to 
identify recommendations to AOs and FEMA for actions to take that increase tmst and actions to 
avoid that decrease tmst. This report describes the process used to develop and validate the tmst 
models and the resulting stmcture and functionality of the models. 
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1 Introduction 


1.1 Background 

The Wireless Emergency Alerts (WEA) service, formerly known as the Commercial Mobile Alert 
Service (CMAS), enhances public safety by providing authorized emergency management agen¬ 
cies (EMAs) with the capability to issue alerts and warnings to mobile communication devices 
(e.g., cell phones) in a designated geographic area. WEA is a component of the Integrated Public 
Alert and Warning System (IPAWS) operated by the Federal Emergency Management Agency 
(FEMA) in cooperation with the Federal Communications Commission (FCC) and supported by 
the Department of Homeland Security Science and Technology Directorate (DHS S&T). 

WEA messages may be initiated by authorized national, state, local, tribal, and territorial EMAs. 
Three categories of WEA messages may be sent: 

1. Presidential — Only the president of the United States can issue a Presidential Alert. This 
message enables the president to alert or warn a specific region, or the nation as a whole, of 
an event of critical importance. 

2. Imminent Threat — EMAs may issue alerts to specific geographic areas affected by an imme¬ 
diate or expected threat of extreme or severe consequences. Threats may arise from a number 
of sources, including weather conditions (e.g., tornadoes, flash floods), law enforcement ac¬ 
tions (e.g., riots, gunfire), fires, and environmental hazards (e.g., chemical spills, gas releas¬ 
es). 

3. Americas Missing: Broadcast Emergency Response (AMBER) — EMAs may issue AMBER 
Alerts for missing or abducted children. 

WEA messages are initiated by the EMAs and transmitted to the IPAWS Open Platform for 
Emergency Networks (IPAWS-OPEN) system using the Common Alerting Protocol (CAP) for¬ 
mat. After authentication and verification, IPAWS-OPEN processes the WEA message and sends 
it to the commercial mobile service providers (CMSPs) participating in the WEA service. The 
CMSPs broadcast the alert from cell towers in the area to all compatible cellular devices. The cel¬ 
lular devices produce a distinctive ringtone, vibration pattern, or both and display the WEA mes¬ 
sage. 

1.2 Problem Statement 

Trust is a key factor in the effectiveness of the WEA service. Alert originators (AOs) and EMA 
management must trust WEA to deliver alerts to the public in an accurate and timely manner. Ab¬ 
sent this trust, AOs will not use WEA. Members of the public must also bust the WEA service. 
They must understand and believe the messages that they receive before they will act on them. 
Clearly, FEMA, the EMAs, and the AOs must all strive to maximize and maintain trust in the 
WEA service if it is to be an effective alerting tool. 

In 2012, DHS S&T tasked the Carnegie Mellon® Software Engineering Institute (SEI) with de¬ 
veloping a WEA tmst model. The purpose of this model was to provide data for FEMA that 
would enable them to maximize the effectiveness of WEA and provide guidance for AOs that 
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would support them in using WEA in a manner that maximized public safety. At a high level, our 
approach to this task was to 

1. build models that could predict the levels of AO trust and public trust in specific scenarios 

2. validate these models using data collected from AOs and the public 

3. execute simulations on these models for numerous scenarios to identify 

a. recommendations to AOs and/or FEMA for actions to take that increase tmst 

b. recommendations to AOs and/or FEMA for actions to avoid that decrease tmst 

Results of this work consist of 

• a detailed technical report (this report) describing the process employed in the development 
and validation of the trust models and the resulting stmcture and functionality of the models 

• Trust Model Simulations for the Wireless Emergency Alerts (WEA) Service, a report detailing 
the scenarios and simulations executed on the tmst models [Morrow 2013] 

• Maximizing Trust in the Wireless Emergency Alerts (WEA) Service, a recommendations re¬ 
port analyzing the results of the simulations and identifying tmst-enhancing practices to be 
employed and tmst-degrading processes to be avoided by both AOs and FEMA [Woody 
2013] 

1.3 Project Environment 

1.3.1 Project Context 

This task was conducted as part of a larger engagement between the SEI and DFIS S&T. DFIS 
S&T had contracted the SEI to perform a number of tasks focused on maximizing deployment of 
WEA. These tasks included 

• development of an integration strategy providing guidance to EMAs and AOs in the adoption 
and deployment of WEA [SEI 2013 a] 

• development of a security strategy providing recommendations for FEMA to manage the cy¬ 
bersecurity of the WEA service and recommendations for AOs to manage the cybersecurity 
of their systems accessing the WEA service [SEI 2013b] 

• development of documents to promote WEA adoption by AOs: 

documentation of a 2011 demonstration of the WEA service in New York City [Trocki 
Stark 2013] 

development and documentation of a collection of best practices addressing AO adop¬ 
tion and use of WEA [McGregor 2013] 
development of a WEA trust model (this task) 

A great deal of synergy existed between these tasks, enabling the SEI to share knowledge and 
resources to the benefit of all tasks. 

1.3.2 Project Team 

The project team consisted of members from the SEI and from SRA International, Inc. The SEI 
team members provided the necessary internal domain knowledge of WEA and supplemental ex¬ 
perience with statistical analysis and probabilistic modeling. The SRA members contributed 
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knowledge of previous workshops aimed at identifying trust factors along with access to an exten¬ 
sive list of emergency alert notification officials from across the United States. 

The SEf was also assisted in this task by numerous EMA organizations, as noted in the Acknowl¬ 
edgements section, that provided both data and valuable insights into public alerting through par¬ 
ticipation in interviews, surveys, and other data collection vehicles. 

1.4 Intended Audience 

As noted earlier, this is one of three reports resulting from this work. This report is not specifical¬ 
ly aimed at members of the alert origination community or at FEMA staff concerned with sustain¬ 
ing and improving the WEA service—that audience will be addressed by the report Maximizing 
Trust in the Wireless Emergency Alerts (WEA) Service [Woody 20i3]. 

This report targets an audience who wants to understand the process of developing and using trust 
models. This audience is likely to include 

• researchers who wish to understand, develop, or use trust models in public alerting, or any 
other domain 

• public alerting researchers who wish to understand, use, or expand on the work contained 
herein 

Therefore, this report addresses only the trust model development process and the trust models 
themselves, ft does not address the use of the models to execute simulations or the findings from 
the analyses of these simulations. These topics are contained in the other reports. 
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2 Challenge 


During the initial planning for this project, we wrestled with the challenge of modeling factors 
related to both the public’s and the AO’s trust in the WEA system. Due to the subjective nature of 
these factors, we concentrated on probabilistic modeling as an attractive method to combine the 
expected historical data from research into emergency notification systems with more subjective 
data derived from expert opinion. The plan that we implemented resulted in a unified model that 
enabled simultaneous evaluation of the effects of many driving factors on outputs representative 
of trust. This construct supported the need to evaluate a wide variety of scenarios to develop guid¬ 
ance to WEA stakeholders regarding events, factors, and actions that may affect frust in the WEA 
service. 

Another challenge arose from the number of factors influencing frust, stressing our ability to con¬ 
fidently cover the space of interactions between factors. These factors could have been assessed in 
literally billions of combinations. We addressed this issue through the use of fractional factorial 
design methods and Monte Carlo simulations. 


CMU/SEI-2013-SR-021 | 4 



3 Scope of Effort 


We identified a myriad of factors related to trust during the review of previous related work on 
effectiveness of emergency alert notification systems. Examples of the wide-ranging factors in¬ 
cluded economic, geographic, cultural, and demographic factors. Upon reflection of these factor 
categories and the mission of this project to develop a quantitative trust model of the WEA system 
as an aid to guide WEA system deployment and use, we decided to target factors that participants 
in the WEA system could directly control or significantly influence. Consequently, we leveraged 
only the previous work related to controllable factors and focused ongoing interviews of emer¬ 
gency alert notification officials and members of the public on potential controllable factors. Ex¬ 
amples of factors excluded from the modeling include gender, marital status, employment status, 
and other personal demographics. 

A form of narrative research from interviews and surveys of EMA officials and the public provid¬ 
ed the bulk of the information and quantitative data employed within the trust model. We priori¬ 
tized causal factors of trust that appeared in both the previous literature and the initial expert 
interviews. Then, we added to the target list additional factors that appeared common to multiple 
expert interviews. We decided to be more inclusive rather than exclusive with potential causal 
factors, knowing that subsequent survey results would identify factors that did not appear to be 
causal in nature to the trust model outcomes. As a result of the above process, we pursued a short 
list of controllable factors in the trust model that served to drive the subsequent survey questions 
used to determine the strength of causal relationships within the model. 

The last scoping decision involved how to quantitatively treat causal factors that did not have 
causal factors of their own. These factors would require a probability distribution representing 
their historical behavior. In the absence of such historical distributions, we decided to implement 
prior distributions of ignorance, such as the uniform distribution. Essentially, the tmst models 
treat these factors with equal chance of any value on the 0-100% scale. As more historical infor¬ 
mation about these isolated factors becomes available, the model may be easily updated to incor¬ 
porate known probability distributions and then used to evaluate different scenarios of factors for 
resulting values of the trust model outcomes. 
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4 Assumptions and Constraints 


The following assumptions and constraints served to evolve the form of, and inputs to, the proba¬ 
bilistic trust model, thereby influencing the results gained from the modeling effort. 

Budget and schedule. During the literature search, we amassed a wealth of information of related 
work on effectiveness of emergency notification systems. It became apparent that the entire budg¬ 
et and schedule for this project could easily be exhausted in an attempt to thoroughly digest the 
voluminous materials and, specifically, the bibliographies provided by Dr. Mileti. Consequently, 
we decided to leverage the materials in a non-exhaustive approach that served to inform the inter¬ 
view questions addressed to the experts and help prioritize the set of factors used to create the 
probabilistic model. Many factors not pursued in this project could still be the subject of study and 
evaluation for causal modeling. However, the expert interviews added the necessary confidence to 
the set of factors identified for continued study in this project. 

Access to emergency alert notification officials. Recognizing that emergency alert notification 
officials had little time to assist with research, as well as the fact that several emergency situations 
occurred in the United States during the conduct of this project, we interviewed as diverse a set of 
officials as possible. The combined efforts of the SEI and SRA staff helped to ensure a balance in 
the expert interviews and provided a rich base of 560 email addresses for the emergency alert no¬ 
tification surveys. 

Access to representative members of the pnblic. We encountered a number of barriers in reach¬ 
ing a reasonably sized sample of members of the public to participate in the trust model surveys. 
Issues of survey representativeness and randomness remained problematic. In Section 11, which 
covers the analysis of the surveys, we discuss challenges regarding external validity and the abil¬ 
ity to generalize from this sample. We finally decided to use over 5,000 email addresses from four 
different sources to reach a semi-balanced and diverse population. The targeted public email ad¬ 
dresses, used with appropriate approvals, were derived from the following four groups: 

1. the ASQ Reliability Division membership 

2. the ASQ Software Division membership 

3. the local Pittsburgh members of INCOSE 

4. staff from within the Software Engineering Institute 

Limitations on data collection from interviews and snrveys. As mentioned earlier, we had to 
rely on interview and survey data rather than observational data. Specific constraints surfaced dur¬ 
ing the conduct of the interviews and surveys related to the need to secure feedback on joint con¬ 
ditional probabilities related to scenarios of multifactor behavior. For example, several initial 
versions of the interview questions and survey questions would have asked respondents to provide 
feedback on scenarios possessing specific behaviors of 7 to 12 factors. The respondents would 
have had neither the cognitive ability nor the patience to provide differentiated responses to ques¬ 
tions with minor nuances. As a result, we decided to primarily focus on 1:1 factor relationships to 
build the probabilistic model, with validation scenarios involving 3 to 5 causal factors serving to 
identify when major interaction effects of factors existed. Aligned with this approach, we adopted 
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an independence assumption among causal factors until sufficient data demonstrated that the as¬ 
sumption was inappropriate. Although specific interaction effects did not appear to surface, some 
degree of interaction occurred, thereby causing moderated values of trust outcomes in specific 
scenarios. As a result, some extreme trust outcomes may be understated. Nevertheless, we ob¬ 
served interesting results across the validation scenarios that will drive useful conclusions in the 
Maximizing Trust in the Wireless Emergency Alerts (WEA) Ser~vice report. 

Need to address diverse seenarios with ineomplete information. Early in the project, we 
learned that the trust model would have to provide results based on limited or incomplete infor¬ 
mation about different causal factors. This specific criteria significantly influenced the decision to 
pursue a Bayesian belief network (BBN), knowing that such models may operate with incomplete 
information. 

Need to ereate a hierarehal model. The relatively large set of causal factors initially identified 
by the literature review and expert interviews, combined with the need to handle incomplete in¬ 
formation of factors, significantly influenced the pursuit a hierarehal model of causal factors. Es¬ 
sentially, instead of having 40 factors all drive a single trust outcome, we decided that the 
research and modeling would be more practical if conducted in a hierarehal fashion. We observed 
that experts could help categorize and group causal factors such that they could evaluate more 
basic “micro” scenarios for strength of relationships. This approach was confirmed during an ex¬ 
ercise in which both SEl and SRA experts participated in a cause-effect matrix analysis of poten¬ 
tial causal factors. The experts found it practical to evaluate the high-priority cause-effect 
relationships and think of scenarios in a hierarehal fashion. In the end, this approach helped im¬ 
mensely with the practical aspects of using AgenaRisk as a probabilistic modeling platform. 

Compliance with human-subjects research requirements. Realizing early in the project that 
subjective ratings of causal factors of trust would be necessary, we submitted the required applica¬ 
tion for Human Subject Research (HSR) and associated HSR materials and received approval 
from the CMU Internal Review Board (IRB), as shown in Appendix A. The HSR requirements 
provided much guidance on how to protect the participants of the interviews and surveys and how 
to collect and secure the resulting data. On the other hand, some of the required introductory lan¬ 
guage and questions prefixed to each survey may have negatively affected the response rates. Alt¬ 
hough we did not quantify this detriment, we received oral feedback from some respondents 
internal and external to the SEl who found the survey too time consuming and bureaucratic to 
take in a voluntary fashion. Additionally, while the response rates were low (7—12%), the number 
of responses provided sufficient data to derive an input distribution of prior probabilities for the 
model. However, the question of bias in the input distribution remains. 
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5 Approach 


During project planning, we developed the project flow depicted in Figure 2. These 14 steps pro¬ 
vided the ability for the SEI and SRA staff to contribute in respective areas of strength, coordinate 
dependencies among activities, and leverage existing work. A short description of each step fol¬ 
lows to include the primary value added to the development of the WEA tmst models. 



Figure 2: WEA Trust Model Development Flow 

Step 1: Plan the Project 

We capitalized on a number of previous experiences to help identify the work breakdown struc¬ 
ture, effort, and schedule of the development of the WEA trust models. Specifically, SEI team 
members brought skills in the application of BBNs from previous work both within and external 
to the SEI. SEI team members also brought skills in the design and issuance of online surveys 
using the Qualtrics tool. Last, SEI team members brought experience from previous submissions 
for human-subject research, thereby facilitating approvals from the CMU IRB, as shown in Ap¬ 
pendix A. The SRA team members brought experience and information from previous workshops 
involving a diverse set of AOs. Both SEI and SRA staff contributed valuable contact information 
for potential survey respondents, thereby eliminating one of the major risks identified for this pro¬ 
ject. 


CMU/SEI-2013-SR-021 | 8 













































step 2: Review the Literature 


As we show in Section 6, a healthy literature review provided a wealth of information to guide the 
development of the trust models. One emphasis during the literature review was to identify previ¬ 
ously recorded, controllable factors affecting trust in public alerting systems. Another special em¬ 
phasis was to identify current research reports’ suggestions for next steps and continued research, 
as this information substantiated and informed the pursuit of a quantitative model. We hope that 
this modeling effort will contribute to the community’s appreciation for such quantitative model¬ 
ing and inspire additional data collection, cognizant of how such information may be readily ap¬ 
plied in an operational model. 

Knowledge from the literature review was then augmented by interviews with public alerting sub¬ 
ject-matter experts, in particular. Dr. Art Botterell and Dr. Dennis Mileti. Dr. Mileti had authored 
a number of research reports focused on factors affecting the public response to emergency notifi¬ 
cations, including two research reports that provided a wealth of knowledge and insight into fac¬ 
tors that appeared appropriate for the trust modeling [Mileti 1990, 2006]. Mileti foretold the need 
for a quantitative trust modeling approach [Mileti 1990]. Among the topics of discussion with 
Mileti was our situational need to depend on interview data and survey data from both members 
of the public and the AO community. While both we and Mileti realized that observational re¬ 
search (measuring actual behavior, possibly under controlled conditions) was a superior way of 
measuring tmst, we also realized that such research was not feasible in this circumstance. We 
chose to proceed with an interview and survey approach, carefully constructing the interviews and 
surveys to minimize the risk of reported results diverging from actual behavior. 

Step 3: Design the Alert Originator Interview Scripts 

The SEl and SRA staff collaborated to create interview scripts consisting of required and optional 
questions as well as follow-up probing questions for the interviews of AOs. Appendix B provides 
these interview scripts. The scripts enabled efficient interviews with time-limited interviewees. 
Most interviews were telephonic, but some interviews occurred in person at the SEI in Pittsburgh. 
We used these interviews to solicit information on tmst factor ideas related to both the public and 
the AO communities. 

Step 4: Conduct Initial Alert Originator Interviews 

We conducted a series of interviews of AOs during the period of October 2012 through January 
2013. Although we led many interviews with most of the team members present, we led a number 
of interviews with only one or two team members present. Generally, each interview lasted be¬ 
tween 45 and 90 minutes, and we often took turns asking baseline and follow-up questions. Sev¬ 
eral of the AOs were gracious in allowing follow-up interviews as we sought to further clarify or 
confirm responses. We were encouraged by how forthright the AOs’ responses were with regard 
to what currently works and does not work within their alert notification systems. Their demon¬ 
strated professionalism added confidence in the quality of responses to the interview questions. 

Step 5: Summarize Interview Notes 

Once all of the interviews were completed, we summarized and condensed the different team 
members’ notes into a single document, shown in Appendix B. This process helped to reduce re- 
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dundancy and conflict among team members’ notes as well as to enrich specific responses based 
on which team member took the most clear and detailed notes. This process also enabled us to 
solve uncertainties related to interviewees’ use of acronyms and terminology often foreign to most 
of the team members. The team discussion of the summary notes also served an educational pur¬ 
pose by enlightening team members new to the emergency notification domain. 

Step 6: Synthesize Initial List of Causal Factors for WEA Trust 

With the interview summary completed, we created an initial list of potential causal factors of 
trast from both the literature review and the interview results, shown in Appendix C. This list 
evolved through a number of iterations of team review to the final list of causal factors for each 
BBN model, also shown in Appendix C. Due to the length of the original list and some overlap or 
slight nuances in definition, we collapsed a number of factors into single named factors as an aid 
to the subsequent surveying and modeling. The final list of factors and definitions for the public 
and AO tmst models appear in Appendices D and E, respectively. 

For the public trust model, we found Mileti’s proposition of a sequential model of Flearing ^ 
Understanding ^ Believing ^ Acting to be a compelling approach to modeling public trust in the 
WEA system. Additionally, Mileti’s report provided an abundance of factors to consider in the 
tmst modeling [Mileti 2006]. 

For the AO tmst model, we decided on three factors that logically fit the decision process of AOs 
in determining whether to use the WEA system: 

1. Available: Is the WEA service available for use by the AO? 

2. Appropriate: Is the WEA service a suitable alert tool for the incident or event? 

3. Effective: Does the AO perceive that the WEA service will be an effective alerting tool? 

As discussed in more detail in Sections 6 and 7, we first studied factors related to tmst identified 
in previous work, and then verified and extended that understanding through subsequent inter¬ 
views and surveys with members of both the emergency alert community and the public. The in¬ 
terviews drove the decision process for choosing the high-priority factors to model, and the 
surveys provided the quantitative basis to link the various factors together and with various trust 
outcomes. 

Step 7: Conduct Cause-Effect Analysis Between Factors 

With the reduced set of prioritized causal factors for each tmst model, we then conducted a cause- 
effect scoring between pairs of factors using a cause-effect matrix popularized in scenario plan¬ 
ning [Lindgren 2009]. We represent the results of these scorings as hierarchies of factors for the 
public and AO tmst models in Appendices F and G. Due to schedule and resource constraints, the 
cause-effect scoring remained subjective and based on the consensus of the team. We then used 
this relationship scoring for the next step of designing the BBN model. 

Step 8: Design the Bayesian Belief Network Model 

Designing a BBN model with cause-effect arrows may be accomplished in many ways. A re¬ 
searcher with access to a wealth of historical data regarding the factors and their relationships 


CMU/SEI-2013-SR-021 | 10 


could use a method such as structural equation modeling to help decide which relationship arrows 
to model [Hoyle 2012]. Alternatively, one could analyze the cause-effect relationship matrix as a 
design structure matrix [Eppinger 2012] and then use a reduction technique to remove any and all 
cycles from the matrix. In this project, the simplicity of cause-effect relationships enabled us to 
informally identify and resolve any cycles in the factor relationships, such that we produced an 
acyclic graph and directly converted it into a BBN. Appendix H shows the resulting BBN for the 
public tmst model, while Appendix 1 shows the BBN for the AO tmst model. These appendices 
depict both the condensed forms of the public and AO trust models with primary factors noted 
and the expanded versions of the public and AO trust models. The expanded models show the 
additional, artificial joining nodes necessary to accommodate the combinatorial effects often en¬ 
countered in BBN modeling. Additionally, operational copies of the actual AgenaRisk model files 
are available to the sponsor as potential assets to be updated and used over time. 

Step 9: Develop Survey Questions to Populate Quantitative Relationships within 
the BBN 

Developing the survey questions to enable populating the BBN relationships quantitatively proved 
to be more challenging than originally envisioned. We originally thought that asking pairs of 
questions with indications of which factors would be set at each “setting” would be too abstract. 
Additionally, we remembered from the discussions with Dr. Mileti that what people say and what 
they do may be entirely different. Consequently, we decided to create survey questions with real¬ 
istic scenario descriptions for respondents to assess. Although this approach still involved a re¬ 
spondent’s answer, it moved the questioning closer to evaluating a respondent’s behavioral 
response. Appendix J includes the surveys for the public trust model while Appendix L includes 
the surveys for the AO tmst model. We developed three surveys for each model after inifial time 
trials confirmed that a single survey would take too much time for a respondent to reasonably fin¬ 
ish without dropping out. Consequently, an analysis of the required number of survey questions 
containing scenarios, along with the number of available email addresses for the public and AO 
communities, produced a design of three surveys for each BBN model. The analysis appears to 
have been valid because the subsequent dropout rates for the surveys were minimal. 

Step 10: Administer Surveys Using Qualtrics 

The SEl administered the surveys using the Qualtrics survey tool, compiling the survey questions 
into questionnaires within the tool. The questionnaires were then piloted among a select group of 
people to ensure the proper operation of the tool, presentation of the questions, and collection of 
the responses. 

Our choice of Qualtrics as a survey tool was fortuitous. The SEl already possessed a she license 
of Qualtrics to conduct surveys, without limits to size or type of survey. Additionally, the ASQ 
required that its internally adopted survey license of Qualtrics be used to afford maximum protec¬ 
tion for members’ email addresses. We thus reaped efficiencies with the painless export and im¬ 
port of surveys between the SEl and ASQ Qualfrics licensed insfallations while mainfaining 
needed confrol and privacy of email addresses. This arrangement further supported the CMU IRB 
requirements to safeguard and control raw information and subsequent data from surveys involv¬ 
ing human-subject research. Qualtrics also provided the ability to control who took the surveys. 
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thereby satisfying CMU IRB requirements that respondents meet specific criteria before they 
would be enabled to take the survey. 

Step 11: Analyze Survey Results; Populate the BBNs 

Once the survey time period closed, we downloaded and analyzed the raw results at the individual 
respondent level for each survey. Only personally de-identified data was shared among team 
members for purposes of review and statistical analysis to feed the BBN models. We provide 
sanitized results of the public trust surveys in Appendix N and sanitized results of the AO surveys 
in Appendix P and R. We summarized the results in two ways: (1) according to the profile re¬ 
sponses against the original 7-point scale and (2) according to a 0-100% probability scale derived 
from mapping the 7-point scale to the 0—100% probability scale. The primary purpose for convert¬ 
ing the ordinal scale of 1—7 to a continuous probability scale of 0—100% was lo enable us to popu¬ 
late the BBN trust models with most factors measured on probability scales. However, the AO 
BBN model and associated surveys maintained both ordinal and nominal factor scales, which 
served a purpose based on the nature of the factors in the AO BBN model. The AO factor settings 
are shown in the AgenaRisk tool export in Appendix T. 

To populate both BBN models, we compared specific groups of questions to help quantify a given 
1:1 factor relationship in the BBN. Appendix K gives the groupings of questions mapped to spe¬ 
cific BBN relationships for the public trust model while Appendix M gives the similar mapping 
for the AO tmst model. The statistical comparisons conducted for the 1:1 factor relationships 
within the public and AO trust models appear in Appendices O and Q, respectively. 

To quantify the relationships in the BBN models, we took the following specific approaches. For 
the public tmst BBN model, the probability of acting on a WEA alert message was conceptually a 
reliability series calculation of the product of the probability of hearing, the probability of under¬ 
standing, the probability of believing, and finally, the probability of acting. Likewise, within the 
AO BBN model, the probability of using the WEA system was conceptually a reliability series 
calculation of the product of the probability that the triggering incident is appropriate for the use 
of WEA, the probability that the WEA system is available, and the probability that the WEA sys¬ 
tem would be effective for the given incident. The intended consequence of the reliability series 
calculation is that each item in the reliability series is required to achieve the overall outcome of 
Acting or Utilization of the WEA System. Traditional averaging methods would not achieve this 
because the averaging calculation could mask low probabilities of specific factors. We present the 
full description of the factor formulas within the public and AO BBN models as automatic Age¬ 
naRisk tool content exports in Appendices S and T, respectively. 

Step 12: Design Validation Scenarios; Conduct Interviews 

The SEI staff members created validation scenarios to use after completing each BBN model as a 
way to verify external validity and to demonstrate the model validity for scenarios in which inter¬ 
action of effects might exist between multiple causal factors. To accomplish this, the SEI staff 
employed fractional factorials from statistically designed experiments to identify a minimum 
number of scenarios that would test the interactions of a set of factors in context of a given inter¬ 
im or final output factor in the BBN. Consequently, we used a set of fractional factorial designs as 
the designs for the validation scenarios. As before, each row in a fractional factorial became the 
basis for a defined scenario in a real-world description that we would evaluate during validation 
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interviews occurring after the BBN model development. Appendix U gives the validation scenari¬ 
os for the public trust model while Appendix Y gives the validation scenarios for the AO trust 
model. 

These scenarios formed the basis for developing validation surveys for the public and AO trast 
models, shown in Appendices V and Z, respectively. Although intended to be as efficient as pos¬ 
sible, the resulting validation scenarios stretched the patience of the participants in the validation 
exercise. The scenarios often had only very slight differences requiring strict attention by the vali¬ 
dation participants. 

To validate the public trust model, we chose a panel of eight representatives of the public at ran¬ 
dom from the SEI staff. Advance knowledge of the WEA service was not a criterion for selection, 
and many of the panel members had minimal exposure to WEA. To validate the AO trust model, 
we interviewed eight emergency management professionals familiar with the WEA service. We 
conducted the interviews either in person or telephonically. 

We provide the results of the validation surveys for the public and AO surveys in Appendices W 
and AA, respectively. These results proved invaluable to validating the BBN models because they 
purposely embodied the joint effects of multiple factors, thereby providing the opportunity to see 
interaction effects on trust outcomes. 

Step 13: Validate Test BBN Models and Adjust 

Once we evaluated the validation scenarios using the validation surveys, we recorded the results 
and compared them to the predictions made by the respective BBN model. One way we decided 
to validate the BBN model adequacy included comparing the BBN model prediction used as an 
independent variable with the validation scenario interview result used as the dependent variable 
within a linear regression exercise. Appendix X includes the results of this validation statistical 
analysis for the public BBN model while Appendix AB includes the results of the validation sta¬ 
tistical analysis for the AO BBN model. Most validation results were statistically significant, with 
adjusted values in the range of 50-90%. 

The major adjustments we made to the models after the validation exercise arose in two areas. 
First, the original approach of combining different factors into a single interim factor consisted of 
equal voting through use of a simple averaging scheme. Flowever, validation results demonstrated 
that such averaging made the BBN prediction insensitive to specific causal factors’ behavior when 
compared to the validation scenario interview result. As a result, we used a weighted average 
scheme in which each beginning factor received a weight equivalent to the inverse of the factor’s 
value. This adjustment produced results more aligned with the validation scenario interview re¬ 
sults. The second adjustment occurred within the AO BBN. Specifically, the AO surveys con¬ 
tained questions concerning the likelihood that the AO would use the system if it were available 
or the likelihood that the AO would use the system if it would be an effective mechanism for the 
specific incident. We concluded that such questions were too abstract for AOs to answer reasona¬ 
bly and consistently. As a result, the probability of Utilization of the WEA system within the AO 
BBN simply became the product of the three probabilities associated with Appropriateness, Effec¬ 
tiveness, and Availability. This adjustment greatly enhanced the sensitivity of the AO BBN to 
specific factor behaviors as seen in the validation scenario interviews. 
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In the final analysis, the validation activity produced regression equations that we used to trans¬ 
form the BBN predictions into the expected probabilities of the outcome factors of each BBN. We 
provide the equations for the public and AO trust BBNs in Appendices X and AB, respectively. 
We could have easily programmed these regression equations in the BBN nodes for the respective 
outcome factors, but we chose to show them separately for transparency to the reader. For exam¬ 
ple, the validation activity of the public trust model resulted in the following formulas, shown in 
Appendix X, in which the mean probability prediction for an alert recipient taking action in re¬ 
sponse to an alert is 

Expected Acting Probability = -6.04 + 11.6 * 103_Acting 
And the mean probability prediction for an alert recipient believing an alert is 

Expected Believe Probability = -6.35 + 12.7 * 1 OO Believing 
We include similar “calibration” regression equations for the AO BBN model in Appendix AB. 

Step 14: Evaluate Additional Scenarios; Write Final Report 

After the validation activities, we then evaluated a significant number of additional scenarios to 
feed conclusions and guidance for WEA system stakeholders and AOs, which we will document 
in the Maximizing WEA Trust report. Scenarios may be evaluated using the executable AgenaRisk 
BBN files provided to the sponsor. We provide helpful information, including tool configuration 
settings required to successfully use AgenaRisk, in Appendix AC. At the end of the project, the 
SEI modeling team members then documented the trust model development journey and artifacts 
for this technical report, providing sufficient detail to motivate subsequent use and updating of the 
model as well as compelling evidence that the BBN models may be used to evaluate additional 
scenarios. 
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6 Literature Review 


6.1 Overview of the Literature 

The literature about trust in emergency response focuses on two areas: conceptualizations (princi¬ 
ples) and case studies. We address these separately and then cover papers and bibliographies. 

Most of the literature focuses on the receivers of the warnings (the general public), although some 
addresses the senders (alert originators). For example, one source asks under what criteria AOs 
accept an alert for distribution. 

Of special note, we benefited greatly from an opportunity to discuss the topic of public emergency 
notification systems, and the state of the literature on such systems, with Dr. Dennis Mileti. Dr. 
Mileti’s insight, as we will discuss later in this report, influenced team decisions regarding the 
scope of the trust modeling as well as specific validity concerns with the original research design. 

6.1.1 Conceptualizations 

In 1990, Mileti and Sorenson reviewed over 200 studies of warning systems and warning response 
schemes [Mileti 1990]. These reviews are presented in a comprehensive and highly cited paper, 
but it predated broad use of mobile phones and the internet, so its technological baseline is dated. 

The paper defines types of hazards and groups hazards according to the time frame of prediction 
(short or long), knowledge about impacts (known or unclear), and whether the hazards are easy or 
difficult to detect [Mileti 1990]. It defines three subsystems of warning systems (detection, man¬ 
agement, and response). One chapter discusses the decision to warn the public, content of a warn¬ 
ing message, dissemination channels, and the necessity to monitor to what extent the message was 
received and heeded. Two chapters discuss, first, dilemmas and adverse consequences of warning 
and, second, technological, organizational, and societal issues such as ethics and a philosophy of 
warning. Influenced by this paper, we emphasized trust factors related to the timeliness of the 
alert, the time window to take action, additional detail surrounding the context of the alert, content 
of the actual alert, and sensitivity to under- and over-alerting the public. 

Chapter 5 of the paper describes the process of receiving the warning message and identifies fac¬ 
tors that would cause the public to heed or ignore the warning, most of which were of specific 
interest to this project [Mileti 1990]. Important characteristics of the warnings themselves includ¬ 
ed the source of the warning (which agency); the channel by which the warning was received; 
message consistency, credibility, accuracy, and understandability; and the frequency of warnings, 
all of which we modeled in this project. Characteristics of the population receiving the warning 
included gender, ethnicity, age, stage of life, family contexts, and individual characteristics such 
as a tendency to fatalism or risk perception and experience or training regarding the nature of the 
warning. We deemed these population characteristics, as discussed later, out of scope for the pur¬ 
poses of this trust modeling project. The paper also debunks a series of popular misconceptions 
about the behavior of the public, belief in which can make warnings less useful and even self- 
defeating. 

Mileti and Sutton created a PowerPoint overview that mitigates the dated aspect of Mileti and 
Sorenson’s earlier work by addressing social media and today’s constant flood of media coverage 
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[Mileti 2009]. We addressed both corroboration in social media and the frequency and repetition 
of alerts in this project. This presentation also offers a separate, large bibliography, describes offi¬ 
cial warning systems, notes myths, and makes recommendations. Myths include panic, starving 
the public of information in the name of simplicity, responses to false alarms, control of warning 
information, and the validity of warnings that the public finds via social media. Both the history 
of false alarms and the ability to corroborate alerts with social media did persist as factors in our 
tmst models. Mileti and Sutton conclude that message content, repetition, cues, and milling are 
very important. Their list of factors that go into an “evidence-based” warning informed a number 
of factors in our models. 

Jackson, Faith, and Willis took an engineering approach to evaluating the reliability of emergency 
response systems [Jackson 2010]. They apply the failure-mode effects and criticality-analysis 
technique from systems engineering to assess what could go wrong with an emergency response 
system, how likely and how consequential the failure would be, and methods of computing a reli¬ 
ability number. Such analyses can vary depending on assumptions about public response and thus 
provide input into trade studies about different warning strategies. This work influenced our deci¬ 
sion to adopt the Available, Appropriate, and Effective trust factors within the AO trust model. It 
also convinced us to adopt a probability-based approach to modeling trust, including a reliability 
series calculation that multiplies probabilities of the sequential parts, effectively treating each as¬ 
pect as essential to the overall function of tmst. 

Aloudat and Michael review how “location-based services” have been used to date in warning 
systems [Aloudat 2011]. Location-based services include “any service that provides information 
pertinent to the current location of an active mobile handset.” For example. Enhanced 911 (E-911) 
calls and notifications can be sent to all cell phones within a geographical location affected by a 
disaster warning. The authors compare Short Message Service (SMS) text message technology 
with Cell Broadcast Service (CBS). SMS messages are sent to one phone at a time, which can be 
very slow. Furthermore, SMS messages are not delivered to phones located within a designated 
area; they are delivered to phones registered within a designated area. CBS broadcasts to all 
handsets within one cell tower area and can be very fast, but the handset must be capable of re¬ 
ceiving and displaying such broadcasts. 

Aloudat and Michael give a brief history of warning networks in the United States, including 
WEA under its former name, the Commercial Mobile Alert System (CMAS) [Aloudat 2011]. In¬ 
formation is provided about other countries ranging from Finland to Malaysia regarding the tech¬ 
nology used, whether carriers participate voluntarily, and whether they are compensated by the 
government. The paper concludes with basic and optional requirements for a location-based 
emergency system. 

Sanquist, Mahy, and Morris discuss the public acceptability of 11 types of implemented security¬ 
enhancing systems and activities [Sanquist 2008]. They measured attitudes along with 14 system- 
rating attributes; attitudes depended mostly on perceived effectiveness and perceived intrusive¬ 
ness. The paper helped identify what factors cause distmst among the public and thus helped 
frame the tmst discussion. They discussed evaluation of trust as a “next step.” From this work, we 
decided to include a number of tmst factors associated with public awareness, perception, rele¬ 
vance, and history of relevance in the tmst models. 


CMU/SEI-2013-SR-021 | 16 


In an International Atomic Energy Association lecture titled “Instructing, Warning, and Informing 
the Public,” Lafortune and Borth describe how to communicate with the public regarding radia¬ 
tion emergencies [Lafortune 2011]. They stress that “Honest, accurate, and timely information 
builds trust.” AOs (or perhaps more accurately, designers of warning protocols) should under¬ 
stand how the media works and use it to maximize the effectiveness of their alerting practices. 
Both trust models in this project also attempted to cover the dimensions of honesty, accuracy, and 
timely information. Additionally, we modeled trust factors related to corroboration in social me¬ 
dia and the degree to which multiple sources were used in the alert. 

The Working Group on Governance Dilemmas in bioterrorism described five strategic goals of 
leadership in the case of deliberate or accidental epidemics: limiting death and suffering, defend¬ 
ing civil liberties, preserving economic stability, discouraging scapegoating, and improving resili¬ 
ence [Schoch-Spana 2007]. Conflicts among these lead to leadership dilemmas. This report 
provides recommendations for addressing these goals while maintaining or growing “social trust” 
based on a factual understanding of myths about public behavior during emergencies. For exam¬ 
ple, public panic occurs in movies more often than in reality; fear of panic is not a reason to with¬ 
hold information. The bust models in this project consequently investigated factors related to a 
recipient’s ability to seek corroboration via information from additional channels and via confir¬ 
mation from other alerting organizations, due to overlap in geographic responsibility. 

McGee and Gow examined university emergency alert systems and reported on what factors lead 
to better adoption of the Mileti warning response process activities (hear, believe, act, personalize, 
etc.) [McGee 2012]. Messages were sent by SMS. Most students heard and understood the mes¬ 
sages, but they were unsure about vaguely described threats and unsure where and how to evacu¬ 
ate. As a direct consequence, we included trust factors related to clarity of the alert message, 
message content and context, including relevance; reason to take action; and the nature of the rec¬ 
ommended action to take. 

Wood, Bean, Liu, Madden, Mileti, and Sutton compared formats of messages, including CMAS 
messages, from 90 characters up to 1,380 and more, to see how people reacted to them [Wood 
2012]. They included a modified form of the Mileti factors under the heading “‘Sensemaking’: 
Understanding, Believing, Personalizing, Deciding, and Searching and Confirming.” We modeled 
a trust factor related to the text length of an alert message in the AO trust model, thereby seeking 
to capture that community’s sensitivity to how the length of the text message impacts perceived 
effectiveness of the alert. 

Brothers and Pavlov modeled public risk perception, events, event media coverage, and response, 
including economic activity, using system dynamics [Brothers 2009]. 

Kapucu focused on factors that make a university more resilient to disaster. The factors on the 
figure on page 24 contributed to this work’s original set of factors [Kapucu 2010]. 

Bums and Slovic modeled a community’s response to a terrorist attack with systems dynamics 
modeling [Bums 2007]. The authors in particular discuss how fear diffuses. Although they do not 
discuss warnings specifically, they provide a good scenario with context for warnings. 

Glantz reported on a workshop held in Shanghai regarding early warning systems, with most of 
the attendees coming from weather and “hydrological” backgrounds [Glantz 2004]. One group of 
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participants focused on issues related to “receiving, believing, and acting on the warning” and 
thus is very relevant to this work. We used the chart on page 56 of this report to help in under¬ 
stand Receiving, Believing, and Acting on warnings. Glantz’s Appendix 1 is a case study of hur¬ 
ricane warnings, prior to their hitting land in Cuba. 

6.1.2 Case Studies 

Most case studies of emergency warning systems accommodate a wide variety of potential emer¬ 
gencies, including weather events such as tornadoes and hurricanes, geophysical events such as 
earthquakes, technological events such as the release of radiation, terrorist acts, and even alerts 
about abducted children. The case studies tend to focus attention on how the news of the event is 
transmitted to members of the public who need to hear it. Some case studies are limited to warn¬ 
ings appropriate to a specific agency, such as just weather [e.g.. Jacks 2010]. 

Pelusco and Michael discuss trust in the sense of ethics rather than in the sense of how to imple¬ 
ment bust in a warning system [Pelusco 2007]. They used scenarios to provoke positive and nega¬ 
tive responses from the participants regarding the ethics of using systems that transmit their own 
locations to other service providers. Both security and privacy risks can result; the authors suggest 
removing any of three driving causes because each reduces the risk significantly. The trust models 
in this project did address security as well as the opt-out rate by the public, which could be moti¬ 
vated by a wide range of reasons. 

Fuller, Abramson, and Sury investigated the trust of communities (different ethnic and minority 
neighborhoods of New York City) in a hypothesized pandemic flu warning [Fuller 2007]. The 
communities had varying levels of bust in the government, particularly the local government, and 
had different concerns. The report cites a need to create clarity (a “brand”) regarding emergency 
messages. Other recommendations were to use “311” as a source of emergency information and to 
increase health department outreach. We modeled the source of an alert (local vs. federal) and the 
degree of public awareness of the WEA system as tmst factors. Although we originally consid¬ 
ered factors such as the public’s general trust in government, we excluded these types of factors 
during a simplifying decision to prioritize focus on factors more directly conbollable by AOs and 
WEA system operators. 

Jacks, Davidson, and Wai extend the ideas of weather-forecasting systems to manmade disasters, 
including a shift from crisis management to risk identification and risk management [Jacks 2010]. 
They describe “nowcasting,” or immediate weather forecasting. Chapter 4 provides a number of 
hazard warning system examples. In general, this article takes a government policy point of view 
and does not describe the trust that the general public has in any warnings. 

Udu-gama discusses the feasibility of implementing a public warning system using cell broadcast 
for the nation of the Maldives [Udu-gama 2009]. Fler table of SMS vs. CBS features cites Aloudat 
and Yan [Aloudat 2007]. Udu-gama discusses systems and mobile providers in the Maldives to¬ 
day and compares their features to the needs for an emergency system. The author details how 
general considerations apply to the Maldives, which consists of multiple archipelagos. 

Faith, Jackson, and Willis studied failure types following 70 representative incidents [Faith 2011]. 
They developed a fault tree that showed different causes for a failure at an emergency operations 
center (EOC), then coded the 70 incidents according to which actual causes occurred. 


CMU/SEI-2013-SR-021 | 18 


Redlener, Abramson, Stehling, Grant, and Johnson survey the readiness of U.S. communities for 
responses to terror, security, and other disasters [Redlener 2007]. Questions included confidence 
in government, ability and willingness to evacuate, personal preparedness, and perceptions of 
community preparedness. Many Americans believe that the threat of disaster is high; however, 
this does not often prompt them to make preparations for disasters. This paper touched on the “be¬ 
lieve” activity: Americans believe weather-related emergencies are more likely than other types 
and tmst the CDC more than FEMA, and FEMA more than the president (G.W. Bush). 

Pemsco and Michael evaluate location-based services regarding control, trust, privacy, and securi¬ 
ty using five connected fictional scenarios to identify security and privacy risks [Perusco 2007]. 
The article focuses on the ethics of the situation. Both kinds of risk could be reduced by ensuring 
that at least one of the contributing factors is small. 

RAND assessed the effectiveness of state and local health departments in communicating about a 
(rare) public health emergency, the FI INI (Swine) flu [RAND 2009]. In general, the states re¬ 
sponded quickly and well. Flowever, only 34% of local public health departments provided infor¬ 
mation within 24 hours, and over half of those responses consisted of links to national sites rather 
than containing any local information. Ringel, Trentacost, and Lurie provided background and 
more discussion about why the local agencies fell short [Ringel 2009]. 

Kapucu made recommendations to improve the disaster-resilience of the University of Central 
Florida [Kapucu 2010]. The figure on page 24 provides factors for the content of alert threats. 

Stanley and Sutton describe types of alert systems in use in different places, how well they work, 
and what issues there might be, including trust issues [Stanley 2011]. They describe uses of social 
media in warnings. They also discuss usability for at-risk subpopulations. They do not present any 
explicit modeling, but their activities are similar to Mileti’s “Flear/Understand/Believe/Confirm/ 
Personalize.” 

6.1.3 Review Papers and Bibliographies 

We used review papers and bibliographies to help us find the sources of factors to include in our 
analysis. 

In their Appendices A and B, Mileti and Sorenson list references by stages of the warning re¬ 
sponse process: factors that influence Flearing, then Understanding, Believing, Personalizing, Re¬ 
sponse, and Confirming [Mileti 1990]. Within each category, both Sender and Receiver factors 
are included. From the Believing factor onward, these are followed by process factors as well. 
Mileti and Sorenson list factors that affect these stages and cite multiple papers that address each, 
including page numbers. They cite a total of 138 papers. 

Mileti and colleagues also distributed a 347-page bibliography (“Annotated Bibliography for Pub¬ 
lic Risk Communication on Warnings for Public Protective Actions Response and Public Educa¬ 
tion, Revision 4”) in 2006 [Mileti 2006]. This includes citations, abstracts, and causal findings, 
arranged alphabetically, one reference per page. 

Bean, Dietz, and Palidwor supplemented the 2006 Mileti work, citing 44 works published after 
2004 [Bean 2012]. The topic was efficacy of warning messages. They follow each citation with its 
abstract, a discussion of method and messages, and findings and implications. 
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Earle reviews studies of trust in risk management, noting the dimensions of trust (intent and abili¬ 
ties; he considers the latter not trust but confidence), its functions (to reduce complexity via social 
risk-taking), and various ways of modeling tmst [Earle 2010]. Earle codes a total of 132 refer¬ 
ences lo trust or confidence, hazard contexts, referents, antecedents, and consequences. 

Sorensen addresses the changes from 1980 to 2000 in prediction and forecasting of hazards, inte¬ 
gration of warning systems, dissemination of warnings, and understanding responses to warnings 
[Sorensen 2000]. Some types of hazard have seen major improvements in prediction or forecast 
(e.g., hurricanes and hazardous material) and in integration of warning systems (e.g., earthquakes, 
nuclear power), but in many cases improvements have been slight. Much has improved in decid¬ 
ing when to tell people to evacuate but not in explaining responses to those warnings. Thirty-two 
factors influence response, but understanding of mechanism is incomplete for many of them, and 
emergency planners can affect only a few of the factors through design of the warning system. 
Sorensen’s paper cites 43 references. Based on this work, we decided to include several trust fac¬ 
tors addressing alert content, context, why a person should act, and the nature of the action to 
take. 

6.2 Literature Search Conclusions 

The literature search, including conversations with Dr. Mileti, provided a solid basis for scoping 
the trust modeling project and to more efficiently design the initial sets of interviews with emer¬ 
gency alert notification staff Combining the literature review and interview results provided a 
more reliable foundation for producing trust models for both the public and the AO communities. 
Additionally, much of the other research in the literature confirmed the need for investigation into 
the quantitative modeling of trust in emergency notification systems. We will amplify this direct 
connection in Section 14, which addresses future work in this area. 
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7 Interview Results 


The initial AO interviews were intended to provide a current baseline of knowledge concerning 
emergency alert notification systems. We recognized that although research literature from the 
past five years was available, the landscape and experiences with notification systems appeared to 
be fluid, as is common with emerging technologies. The interview questions, which we reproduce 
in Appendix B, were thus motivated by a sense of exploring what appears to be working versus 
not working. Additionally, the interview questions sought to anticipate future needs and concerns 
from the AOs’ perspectives, thereby modeling factors that may become significant with respect to 
trust in the WEA system. 

To accomplish this, we selected AOs not only for their experience but for their recent experience 
with emergency notifications. The AOs interviewed for this project comprised federal, local, civil, 
and academic organizations (see the Acknowledgments for a partial list). We provide a condensed 
summary of the interview notes in Appendix B. 

From the common themes within the interview notes, an initial list of potential causal tmst factors 
emerged, which we include in Appendix C. We subsequently analyzed these factors and reduced 
them to a set that drove the probabilistic modeling of tmst along with surveys to help quantify the 
strength of relationships within the probabilistic models. 
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8 Basis for Modeling 


Probabilistic modeling, specifically Bayesian belief network (BBN) modeling, first appeared op¬ 
portunistic due to the need to model subjective expert opinion. A current reference by Fenton and 
Neil proved useful and timely for this work as it is a companion to the AgenaRisk modeling soft¬ 
ware employed for the BBNs [Fenton 2013]. Although Fenton and Neil provide excellent contex¬ 
tual information for BBNs instantiated with the AgenaRisk tool, Martz and Waller adeptly 
summarize the key reasons and benefits of using BBNs as follows [Martz 1991]: 

• BBNs perform objective and subjective data modeling. BBNs are probabilistic models that 
characterize factors with probability distributions. The probability distributions are often sub¬ 
jective in nature, reflecting degree of belief from a domain expert. These models incorporate 
the concept of combining prior knowledge of a factor with current observational data or ex¬ 
pert judgment to produce an updated assessment of the factor. This modeling represents a su¬ 
perset of traditional statistical modeling from the standpoint that in the absence of prior 
knowledge (e.g., using a non-informative uniform probability distribution), the calculations 
yield results similar to traditional statistical analysis. For the WEA trust modeling, the ability 
to model subjective factors combined with the freedom to include any objective information 
provided the robust modeling platform to meet the customer’s needs. 

• BBNs operate with incomplete information. Traditional statistical modeling, such as re¬ 
gression modeling, usually requires knowledge of all the modeled factors before formulating 
a prediction. BBNs, on the other hand, are adept at formulating predictions with one or more 
of the factors left unknown or unobserved. This specific aspect provides needed flexibility in 
evaluating many possible scenarios from a trust perspective. Additionally, real-life scenarios 
often have incomplete or missing data, whether rooted in data collection shortcomings or in 
data misreporting. 

• BBNs predict forward. Similar to traditional statistical modeling, BBNs can predict forward, 
disregarding whether time or logical dimensions apply. 

• BBNS diagnose backward. Contrary to traditional statistical modeling, BBNs can simulta¬ 
neously diagnose backward, for example, to explain the likely conditions that preceded the 
current situation or given outcome. To be more specific, as new evidence or observations are 
made known to a BBN, the BBN will propagate updates in all directions to the unknown or 
unobserved factors. 

• BBNS evaluate unprecedented scenarios. Because of a combination of the above strengths, 
BBNs are capable of evaluating unprecedented scenarios. Specifically, Martz and Waller ad¬ 
vocate Bayesian analysis for reliability modeling so that researchers can still model and eval¬ 
uate unprecedented failure modes, which have no failure data [Martz 1991]. 

• BBNs support learning mechanisms. BBNs inherently can accommodate a learning mecha¬ 
nism not unlike that of neural networks. A stream of new evidence and observation may be 
fed into a BBN, with learning occurring via the use of updated prior and likelihood probabil¬ 
ity functions. Learning Bayesian mechanisms may be most popularly seen in email spam fil¬ 
ters and in Kalman filters used for electronic systems. 
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Bayesian analysis has experienced a roller-coaster history of embracement and rejection, as rec¬ 
orded in a history of Bayesian analysis by McGrayne [McGrayne 2011]. Critics often complain 
that the incorporation of subjective data remains intolerable and subjects the analysis to gaming. 
On the other hand, modem statisticians, such as Kmschke, not only defend Bayesian analysis but 
now proclaim Bayesian analysis superior to null hypothesis testing (NHT) and argue for the im¬ 
mediate cessation of NHT as a statistical tool [Kmschke 2010]. Disregarding how the reader may 
view Bayesian analysis, the SEI staff remain convinced that BBNs are but one of many tools in 
the quantitative toolkit (statistical, probabilistic, simulation) that should be used in a situational 
manner. The next section will provide more detail on the use of BBNs for the WEA tmst model¬ 
ing, thereby increasing confidence in such use of BBNs. 
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9 Development of the Trust Models 


The development of the two BBN trust models occurred in Steps 7 and 8 of the 14 steps described 
in Section 5 of this report. Once we identified the short list of causal trast factors (see Appendices 
D and E), the next step (Step 7) was to decide what the relationships were among the causal fac¬ 
tors and between the causal factors and the trust outcomes. For the public trust BBN, the primary 
trast outcomes included the probability of Flearing the alert, the probability of Understanding the 
alert, the probability of Believing the alert, and the probability of Acting on the alert. For the AO 
trast BBN, the primary trast outcomes were the Availability of the WEA system, the Appropri¬ 
ateness of using the WEA system, and the perceived Effectiveness of using the WEA system. In 
the absence of a wealth of historical data pertaining to these factor relationships, which could 
have been analyzed with structural equation modeling techniques [see Floyle 2012], we conducted 
a cause-effect matrix analysis, leveraging a scenario planning technique from Lindgren and 
Bandhold [Lindgren 2009]. Essentially, we evaluated all of the possible one-to-one relationships 
between factors and assigned a strength score for each directional relationship of cause and effect. 

Upon completing that exercise, we proceeded to break any cycles in the set of factors, knowing 
that we would need an acyclic network for developing the BBN. The resulting hierarchal list of 
factor relationships appears in Appendices F and G. We did not need to break iterative cycles and 
feedback loops among the factor relationships within the AO BBN, as the factors were so com¬ 
pletely distinct in their relationships. Flowever, the public BBN did possess a number of factors 
that simultaneously influenced the different steps of hearing, understanding, believing, and acting 
on an alert message. We evaluated the impacts of arbitrarily breaking cycles in the public BBN 
and remained confident that the arbitrary breaking of the cycles had minimal impact on the out¬ 
come nodes. This is because the nature of weak versus strong factor relationships and the nature 
of the hierarchal design of the BBN cause the broken path to be relatively distant from the out¬ 
come factors in the BBN. 

Before discussing the internal mechanisms of the public and AO BBNs, a quick discussion of 
BBN models is warranted. As described by Fenton and Neil, BBNs consist of a set of nodes rep¬ 
resenting factors in the model [Fenton 2013]. Some nodes may represent factors that are strictly 
either causal or effect in nature while other nodes may represent factors that are both causal and 
effect in nature. For example, as may be seen in the AO BBN model in Appendix I, the node la¬ 
beled Training represents a factor that is causal on the factor Available but is also an effect of the 
combined factors of Skills & Competencies, Understanding, and Practice. In this example, the 
factors Skills & Competencies, Understanding, and Practice are strictly causal factors. The only 
strictly effect factor in this BBN model is the final outcome factor represented by the node labeled 
WEA Utilization. As the reader most likely has surmised by now, the arrows connecting the nodes 
in the BBN represent directional cause-effect relationships. In some cases, the cause-effect rela¬ 
tionship may be weak or viewed as an indirect, influencing relationship or leading indicator rela¬ 
tionship. Flowever, as Fenton and Neil point out, BBNs can be more easily portrayed and 
communicated if the arrows do represent cause-effect relationships [Fenton 2013]. 

Factors within a BBN may possess any of the possible measurement scale types to include nomi¬ 
nal, ordinal, interval, or ratio. Nominal factors would be viewed as having discrete states or levels. 


CMU/SEI-2013-SR-021 | 24 



such as the factor Public Awareness in the AO BBN (see Appendix I). The Public Awareness fac¬ 
tor has two states or levels consisting of (1) the public was previously informed of the WEA sys¬ 
tem and (2) the public was not previously informed of the WEA system. Other factors in the 
BBNs are ordinal in nature such as the AO BBN factor called Alert Frequency. Alert Frequency 
has an ordered set of states consisting of (1) several alerts in the past week, (2) several alerts in 
the past month, and (3) several alerts in the past year. Finally, other factors reflect a continuous 
scale (interval and ratio), such as the factor Effective in the AO BBN model. The factor Effective 
possesses a continuous measurement scale of 0-100, representing a probability score. All of the 
continuous factors in both BBNs represent probability scales from 0% to 100%. 

Appendices S and T depict the measurement scales, states, and foundational formulas used to de¬ 
rive the factors within the public and AO BBNs, respectively. Another nuance related to the in¬ 
stantiation of factors within the AgenaRisk BBN models concerns the continuous factors. Agena- 
Risk enables continuous factors to be declared simulation variables while other continuous 
variables are not simulation variables. The simulation variable distinction enables AgenaRisk to 
conduct more efficient processing of Bayesian propagation algorithms; therefore, we used it when 
possible. Fenton and Neil provide additional information on the nature and use of simulation vari¬ 
ables within AgenaRisk [Fenton 2013]. 

We show the public tmst BBN model in both condensed and expanded format in Appendix FI. 

The condensed format depicts only the cause and effect factors related to the public tmst of the 
WEA system. The expanded format depicts the additional synthetic nodes required in the devel¬ 
opment of BBNs to reduce the combinatorial explosion of factor relationships. See Fenton and 
Neil for guidance on the approach of using synthetic nodes [Fenton 2013]. In the public tmst BBN 
model, the four primary outcomes of tmst involve the probabilities of Flearing, Understanding, 
Believing, and Acting on the alert. Several other factors are modeled as hybrid cause and effect 
nodes, to including (1) Alerts Viewed as Spam, (2) Opt Out Rate, (3) Confirmation by Social Me¬ 
dia, and (4) Relevance of the Alert. These four factors are influenced by other factors and then, in 
turn, influence the primary outcome factors listed above. 

We show the AO tmst BBN model in both condensed and expanded format in Appendix I. In sim¬ 
ilar fashion, the condensed format depicts only the cause and effect factors related to the AO tmst 
in the WEA system while the expanded format depicts the additional synthetic nodes to handle the 
combinatorial explosion of relationships. The primary outcome of the AO BBN model is the 
probability of WEA Utilization by the AOs. The additional outcomes of tmst include the Appro¬ 
priateness, Effectiveness, and Availability of the WEA system. Each of these additional outcomes 
is influenced by separate factor sets comprised of 7 to 12 individual factors. As opposed to the 
public tmst model, which incorporated predominantly continuous factors measured on a probabil¬ 
ity scale of 0-100%, the AO factors are predominantly measured on nominal or ordinal scales. 

We give the specific states or values of these factors in Appendices S and T, along with the for¬ 
mulas used within the BBN. For example, the node labeled Geographic Breadth contains four 
states describing the different geographic situations of WEA system coverage for a given alert as 
follows: 

70% outside zone 

50% outside zone 

30% outside zone 


CMU/SEI-2013-SR-021 | 25 


10% outside zone 


There are several ways to execute the BBN models for purposes of this research. The first use 
case comes from the public BBN model, as shown in Figure 3 to Figure 8. It consists of the ability 
to evaluate a given scenario that comprises observed or hypothesized values for a number of fac¬ 
tors followed by assessing the impact on an outcome factor, such as the probability that an indi¬ 
vidual will take action based on a WEA alert. 

To begin with. Figure 3 depicts the probability expectation for Acting on a WEA alert with no 
specific knowledge of the state of any other factors. In this example. Figure 3 shows that the range 
of the 25th percentile to the 75th percentile of the probability of Acting is 5-12%. The reader 
should also note that the BBN tool does not understand that negative values of probability are not 
allowed, as shown in Figure 3, which depicts a mean and standard deviation that would obviously 
include negative probability values. 



Figure 4 and Figure 5 show how one can now use AgenaRisk to enter an “observation” or pre¬ 
sumption of a setting for the Relevance factor. In this example. Relevance is a continuous factor 
representing the probability that the WEA alert is relevant to the individual. A 0 in this example 
implies there is no relevance of the WEA alert to a recipient. Alternatively, a 100 would have im¬ 
plied a certainty of relevance of the WEA alert to the recipient. 
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Figure 4: Using AgenaRisk to Enter a Presumption for the Reievance Factor 



Figure 5: Entering 0 impiies There is No Reievance of the WEA Aiert to a Recipient 
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As may be seen in Figure 6, we set three other factors to a probability value of 0. This example 
consequently models the scenario in which we set all four factors (Relevance, Action to Take, 
Lead Time Provided, and Time Window to Act) at their most negative settings related to trust in 
the WEA system. Upon completion of the AgenaRisk simulation, the resulting probability of Act¬ 
ing on this WEA alert changes slightly to a 25th percentile to 75th percentile range of 5-11%, as 
shown in Figure 7. Although this appears as a minor change in the probability assessment, the 
validation activity for this model produced a calibration equation that we will discuss in Section 
13. We must apply the calibration equation to the BBN model prediction of probability to arrive at 
the expected probability; therefore, the calibration equation may depict a larger change in proba¬ 
bility than the raw values from the AgenaRisk simulation. Indeed, the validation activity of the 
BBN models did uncover a number of factors that have very little impact on interim and final trust 
outcome factors. Flowever, we retained all factors in both of the BBN models to enable the reader 
to observe that the data from the surveys did differentiate between significant and nonsignificant 
factors as drivers of trust. 



CMU/SEI-2013-SR-021 


28 
















Figure 7: Probability of Acting on the WEA Alert Changes to a 25th-75th Percentile Range of 5-11% 

A second capability of the AgenaRisk BBN models consists of constructing a sensitivity chart 
showing the factors in priority order that influence a given interim or final outcome factor. This is 
useful from the standpoint that within any given scenario, AgenaRisk can report which factors are 
most influential on an outcome factor. This type of information could serve to guide follow-on 
probing questions of domain experts when discussing and analyzing different potential scenarios 
of cascading change drivers. We could also use this information to help assess the highest priority 
risk factors and improvement factors within a given scenario. As shown in the sensitivity chart in 
Figure 8, the outcome of Acting on a WEA alert is most influenced, as expected, by the probabil¬ 
ity that the alert is Believed, followed by the probability that the alert is Understood, followed by 
the probability the alert is Fleard, and last, the probability that the alert contains specific infor¬ 
mation on Who Should Take Action. 


•25 
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Figure 8: Factors Most Influential on the Outcome of Acting 
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A third capability of the AgenaRisk BBN model may be considered a variant of the second capa¬ 
bility previously discussed. The third capability involves the ability to determine the settings of 
“upstream” factors that most likely explain a given setting of an interim or a final trust outcome. 
This may be helpful in answering questions about which likely scenarios of factors lead to a given 
value or setting of a trust outcome factor. This capability could prove most useful when dealing 
with a situation requiring real-time and quick feedback to diagnose what led to the current situa¬ 
tion, possibly enabling immediate mitigating actions. 

In summary, we use the BBN models to evaluate “what if’ scenarios, understand the most signifi¬ 
cant factors of specific trust outcomes, and diagnose what likely events led to a specific trust out¬ 
come. 
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10 Development of the Trust Surveys 


With the trust factors established in Step 7: Conduct Cause-Effect Analysis Between Factors, and 
the cause-effect relationships established in Step 8: Design the Bayesian Belief Network Model, 
we constructed a survey questionnaire that we could use to measure these relationships. For each 
of the arrows in the BBN network, we crafted a question or series of questions that enabled as¬ 
sessment of the strength of the indicated relationship. 

For example, in the AO trust model. Urgency is a factor contributing to the Appropriateness of 
WEA as an alerting solution. In other words, an event must be sufficiently urgent for an AO to 
consider WEA as an appropriate alerting solution. We created a series of questions regarding the 
appropriateness of WEA for varying levels of urgency, as shown in Table 1. 

Table 1: Survey Question Development 



Question responses were on a seven-choice Likert scale: 

• Definitely Not 

• Very Probably Not 

• Probably Not 

• Maybe 

• Probably 

• Very Probably 

• Definitely 

Comparing the response distributions across this set of questions provided a measure of the 
strength of this relationship. 

Question sets of this nature were created for each relationship in each of the BBNs. This resulted 
in a collection of 58 questions for the public trust model questionnaire and 36 questions for the 
AO trust model questionnaire. Piloting of these questionnaires revealed that the time required to 
complete the questionnaires was excessive, at more than 15 minutes per survey. Such a lengthy 
questionnaire would have negatively impacted the response rate. Hence, we divided each ques¬ 
tionnaire into three parts, with each part sent to one third of the total sample. Piloting of these re¬ 
duced questionnaires yielded an average response time of approximately 5 minutes, which we 
thought was acceptable. 
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Appendix J reproduces the resulting surveys for the public trust model, while Appendix L repro¬ 
duces the surveys for the AO trust model. We also mapped the survey questions to the specific 
factor-factor relationships within each BBN model. The mapping between the public trust surveys 
and model appears in Appendix K, while the mapping between the AO trust surveys and model 
appears in Appendix M. 

We used the Qualtrics survey tool to issue the surveys for this project. The surveys of the SEI 
staff and the AOs were conducted through the SETs licensed installation of Qualtrics while the 
surveys of the ASQ Software and Reliability divisions were conducted using the respective ASQ 
division’s licensed installation of Qualtrics. In this manner, the ASQ membership email addresses 
were not released outside of the ASQ domain, in keeping with their existing policy on the control 
and usage of their membership email lists. The SRA team members provided a list of 560 email 
addresses of emergency alert notification staff from federal, state, and local organizations across 
the United States. SRA developed this list from previous workshops and correspondence with the 
emergency alert notification community as recently as the fall of 2011. 

Table 2 characterizes the target audience for the public tmst model surveys; it consisted of indi¬ 
viduals from four sources in which email addresses were accessible within the required schedule. 
The figure also shows the approximate number of invited respondents and the actual number of 
respondents. Due to incomplete responses, the corollary analysis of the surveys may depict slight¬ 
ly different sample sizes. The very low response rates within the ASQ Software Division mem¬ 
bership occurred due to a last-minute decision by the ASQ Software Division to forego use of the 
Qualtrics tool email utility and, instead, deliver the survey invitation and survey link embedded 
within a monthly electronic newsletter. We expected this to cause a precipitous drop in response 
rate but, nevertheless, the ASQ Software Division remained seriously concerned about the volume 


of email going to their members that particular month. 
Table 2: Public Survey Statistics 



Survey 1 


Survey 2 


Survey 3 


Total 



Source 

Sampled 

Responded 

Response % 

Sampled 

Responded 

Response % 

Sampled 

Responded 

Response % 

Sampled 

Responded 

Response % 

ASQ Reliability Division 
members 

789 

68 

9 

789 

105 

13 

789 

79 

10 

2,367 

252 

11 

ASQ Software Division 
members 

689 

0 

0 

689 

0 

0 

689 

2 

0 

2,067 

2 

0 

SEI staff 

196 

25 

13 

205 

26 

13 

200 

28 

14 

601 

79 

13 

Local INCQSE mem¬ 
bers 

34 

2 

6 

30 

4 

13 

28 

3 

11 

92 

9 

10 

Totals 

1,708 

95 

6 

1,713 

135 

8 

1,706 

112 

7 

5,127 

342 

7 


Table 3 characterizes similar data for the AO surveys. Most evidently, the overall response rate 
for the AO surveys (12%) was almost twice the response rate of the public (7%) in keeping with 
the passionate focus that AOs have on this topic as compared to general members of the public. 
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Table 3: Alert Originator Survey Statistics 


Source 

Survey 1 

Survey 2 

Survey 3 

Total 

Sampled 

Responded 

Response % 

Sampled 

Responded 

Response % 

Sampled 

Responded 

Response % 

Sampled 

Responded 

Response % 

Alert Originators 

168 

18 

11 

193 

31 

16 

199 

17 

9 

560 

66 

12 
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11 Analysis of the Surveys 


11.1 Limitations of the Surveys 

We remained sensitive to a number of issues involved in creating successful surveys. This section 
addresses some of these issues and describes how we decided to handle each issue. 

Data collection method. We decided to use multiple-choice surveys as a means of gathering 
quantitative feedback on specific tmst scenarios. This approach met the schedule and budget con¬ 
straints of the project. Given more time and budget, we could have asked open-ended questions 
along with multiple-choice questions in an effort to explore the respondents’ thinking and learn 
more about specific tmst factor interactions. 

Respondent effort. We were sensitive to how long a given interview could last or how many 
questions a survey could have to keep a respondent engaged in providing accurate answers. As a 
result of this analysis, we divided the total set of questions for each BBN tmst model into three 
different surveys, which resulted in very low dropout rates for the surveys. 

Question wording. We knew that terminology would be critical with the survey questions. As a 
result, we avoided most acronyms and cryptic terms used in the emergency notification domain 
and used brief but concrete terms describing alert situations. Piloting the wording of survey ques¬ 
tions with internal SEl staff provided invaluable assistance in rewording problematic survey ques¬ 
tions prior to use with the thousands of planned survey recipients. 

Order. We considered the ordering of questions as a factor to evaluate. Although the Qualtrics 
survey toolset allowed for the randomization of question order, we decided against such extreme 
measures and manually reviewed each survey for issues with question order and question relation¬ 
ship. In some cases, we moved highly related questions to separate surveys. 

Format. Although we initially sought to include cell phone screen pictures within the survey 
questions, the limitations of the Qualtrics tool to use only Rich Text Format for questions led to 
the creation of textual WEA alerts with nuances highlighted with bold and font-size changes. 
Again, the internal piloting of the questions with SEI staff provided many instances of improve¬ 
ment regarding question formatting. 

Structure. We kept the stmcture of the surveys simple to a linear list of questions without logical 
jumping. The only conditional flows in the surveys related to the early questions required as part 
of the human-subject research guidelines. If respondents answered in the negative to any of the 
questions discussed in Appendix A, the Qualtrics tool would then flow immediately to a thank 
you screen before exiting the survey. We did highlight to the respondent in advance the conse¬ 
quence of answering in the negative for any of the three human-subject research questions so that 
respondents would not find themselves refused from taking the survey by mistake. 

Visual layout. We made a concerted effort to reduce, if not eliminate, required scrolling by a re¬ 
spondent within the web browser window during the survey activity. Ideally, respondents would 
take the survey mostly by mouse clicks so that the survey time could be reduced accordingly. 
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We remained vigilant in addressing both the validity (e.g., accuracy) and reliability (e.g., con¬ 
sistency) of the surveys. We addressed the following aspects within the context of overall threats 

to validity [Campbell 1963]: 

• Sample representativeness. Controlled access to respondents’ email addresses heavily influ¬ 
enced the sampled group of respondents, so follow-up on nonrespondents could occur. There¬ 
by, the survey did not have a completely self-selected sample. However, the respondent 
sample taking the public trust survey were admittedly biased from the U.S. population socio¬ 
economic status, since most respondents were members of the ASQ Reliability Division. This 
bias warrants scrutiny in follow-on research by ensuring the polling of a sample with educa¬ 
tional backgrounds and socioeconomic status more representative of the U.S. population. 

With regard to the AO population, we deemed the sample to be representative, as the email 
addresses came from a collection of AO conference and workshop venues from across the 
country. 

• Survey design. We made every attempt to design the survey questions such that they resem¬ 
bled a real-life experience of an emergency alert notification. Consequently, we believe the 
questions did achieve the desired measurement and assessment. However, one specific aspect 
of the questions included a concern for the public’s reaction to alert messages that might con¬ 
tain spelling or grammatical errors. We incorporated errors into the scenario messages in the 
survey, but many of our respondents assumed that we made the error in constructing the sur¬ 
vey question rather than that the error truly occurred in a WEA alert. As a result, we are less 
confident about the results depicting little impact on WEA trust due to spelling and grammat¬ 
ical errors. This aspect warrants further attention in subsequent research. 

• Face validity. We expended a significant amount of effort creating hypothetical emergency 
alert scenarios and actual messages for use in the surveys to help ensure face validity. We be¬ 
lieve the realism in the messages came as close as one could come to conducting observation¬ 
al research of real-life situations. 

• Content validity. Although we did not conduct a statistical test of the content validity, we 
felt that the survey questions were concise and crisp, so that the specific trust factor(s) within 
the question were dominant and not masked or otherwise confused with other factors or side 
issues. In a further attempt to address content validity, we bolded and increased the font of 
specific words in each question to help the respondent realize the nuanced difference between 
similar questions. The internal testing of the questions within the SEI surfaced the confusion 
and challenge of answering the questions properly without the bolding and larger font. 

• Internal validity. To increase confidence that the survey questions can really explain the out¬ 
come we want to research, our survey design included pairs of questions for most cause- 
effect relationships in the BBN trust models. For example, if we hypothesized that a given 
factor (prior knowledge of WEA) influenced belief in the WEA message, we asked one sur¬ 
vey question with a scenario involving significant prior knowledge of WEA followed by a 
second question with a scenario involving little to no prior knowledge of WEA. 

• External validity. To maximize the extent to which we could generalize the results to the 
target population, we identified multiple subpopulations to canvas with the public trust sur¬ 
veys. We intended to analyze the external validity statistically by compared results across 
subpopulations, but the sample sizes precluded this test. Follow-on research efforts should 
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further test the external validity by identifying a completely different set of subpopulations to 
canvas with the surveys. 

From a reliability standpoint, we were concerned with the consistency of measurement using the 
surveys. Although not studied statistically as a rater reliability or repeatability/reproducibility ex¬ 
ercise, the internal iterations of the survey questions under review by the SEI staff provided in¬ 
formal feedback on reliability. This aspect of the above-mentioned additional statistical tests 
would be the most practical to introduce in any follow-on modeling work. 


11.2 Survey Descriptive Statistics 


We provide the descriptive statistics related to the public surveys in Appendix N and the corollary 
material for the AO surveys in Appendix P. In both appendices, we include the original ordinal 
survey results followed by the conversion to a continuous scale of probability. We accomplished 
this conversion with the arbitrary midpoints of each range identified at the beginning of each sur¬ 
vey and recreated here for convenience: 


Definitely Not 

Very Probably Not 

Probably Not 

Undecided 

Probably 

Very Probably 

Definitely 


I would respond to the situation less than 5% of the time 
I would respond to the situation 5-20% of the time 
I would respond to the situation 20^0% of the time 
I would respond to the situation 40-60% of the time 
I would respond to the situation 60-80% of the time 
I would respond to the situation 80-95% of the time 
I would respond to the situation more than 95% of the time 


11.3 Analysis of the Surveys 

We show the analysis of the public trust surveys in Appendix O and the analysis of the AO trust 
surveys in Appendices Q and R. Of special note. Appendix O shares the comparative tests for 20 
different relationships within the public trust model. Of the 20 relationships tested, only 7 had low 
p values indicating significantly different influences on an outcome factor based on an originating 
causal factor. Thus, for the public model, a minority of the cause-effect relationships had strong 
statistical differentiation of outcomes based on causal factor behavior. As a result, a small subset 
of causal factors drives the differentiation of outcome probabilities of Hearing, Understanding, 
Believing, and Acting. The Trust Model Simulations for the Wireless Emergency Alerts (WEA) 
Ser~vice and Maximizing Trust in the Wireless Emergency Alerts (WEA) Service reports will cover 
these relationships in further detail. Appendices Q and R share the statistical comparisons related 
to relationships within the AO trust model. The comparisons here were counter to the public trust 
comparison tests. For the AO trust model, of the 36 comparative tests performed, 29 demonstrated 
statistically significant behavior of outcome factors based on changes to the causal factors. As a 
result, the AO trust model has more capability of depicting differentiated probability outcomes 
based on different scenarios of causal factors than the public tmst model. From a research stand¬ 
point, this difference in survey outcomes may be rooted in the nature of the respondents. We 
would expect AOs to be more aware and sensitive to causal factors within the tmst model than 
average members of society. However, further work in this area should include consideration of 
these noted causality differences. 
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12 Quantified Modeiing Reiationships 


To quantitatively define most relationships in the BBNs, we used a pair of questions that together 
elicited information about the outcome factor based on the two extreme settings of the preceding 
causal factor. This approach kept the elicitation process plausible within the given schedule and 
budget constraints and the combinatorial explosion of multiple causal factors jointly driving a 
common outcome factor. We decided that we could best approximate the joint conditional proba¬ 
bility of a given outcome factor by a weighted combination of the individual relationships of 
causal factors. Anticipating that this approach could overlook important interaction effects of mul¬ 
tiple causal factors on an outcome factor, we decided to design validation scenarios that would 
provide the opportunity to surface any significant interaction effects. As we show in Section 13, 
the statistical analysis relating the BBN outcome factors to the validation scenarios provided a 
transformation function intended to account for the interaction effects. 

During initial model prototyping, we learned that a simple weighted average approach to combin¬ 
ing information of multiple causal factors proved insensitive to the effects of a single causal factor 
on the outcome factor. The effect of averaging 7—12 causal factors masked the individual impacts 
expected of single causal factors. As a result, we decided to employ a weighting scheme using the 
inverse of the factor’s value to weight each causal factor. Thus, causal factors with smaller proba¬ 
bility values would have a greater impact on the determination of the probability of the outcome 
factor. Trial scenarios of one or two causal factors with low probability values combined with 
three to five other causal factors with high probability values demonstrated that this approach 
added sufficient sensitivity to the BBN model outcomes. 

Further research is still warranted to determine a more optimal approach to combining multiple 
causal factors to drive a single outcome factor. Additional research should focus more closely on 
how SMEs and members of the public consider multiple items of information when determining 
whether to understand, believe, and act on WEA alerts. Without expending more energy on this 
particular line of research, we decided that the inverse weighting scheme provided the needed 
amount of sensitivity to serve as a useful model of trust for this project. 
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13 Validation Interviews 


After creating the BBN and using survey data to calibrate it, we initiated a validation process for 
the resulting models for the purpose of ensuring that they provided a sufficiently accurate repre¬ 
sentation of reality. 


For the AO BBN, we created scenarios to exercise 18 trust factors of the model. We gave each 
factor a binary attribute; for example. Training was either sufficient (+) or insufficient (—), and 
Availability was either high (+) or low (—). Clearly, it was impractical to develop scenarios to ad¬ 
dress all combinations of trust factor values; this would have resulted in 2*^ test cases. Instead, we 
chose to categorize the factors in seven groups, as shown in Table 4. Within each category, we 
used fractional factorial methods from statistical design of experiments to develop a series of test 
cases addressing the factors of each category. This resulted in the seven scenarios encompassing 
the 84 test cases shown in Table 4. Appendix Y provides the actual validation scenarios. 

Table 4: AO Trust Model Validation Scenarios 



Availability 


Accessibility 


Reliability 


Ease of use 


Timeliness 


Understandability 


Accuracy 


Urgency 


Severity 


Certainty 


Geographic breadth 


Time of day 
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Based on these test cases, we developed a validation questionnaire that solicited the respondents’ 
actions for each of the test cases. Appendix Z reproduces the questionnaire. 

We contacted nine public alerting SMEs to validate the model. We interviewed each using the 
questionnaire; captured the results; and analyzed them to identify the mean, median, and variance 
of the response. 

We input these same scenarios into the BBN and compared the results with the SMEs’ responses. 
The graphical results of the validation surveys appear in Appendix AA. For example. Case 5 
shows a pronounced difference in AO response between a scenario of high urgency, high severity, 
and low certainty as compared to a scenario of low urgency, low severity, and low certainty. We 
show the subsequent statistical analysis of the AO validation activity in Appendix AB, which 
comprises 

• a table of data with pairings of the BBN model predictions versus the validation interview 
results 

• a scatterplot of the pairings depicting a rough linear relationship 

• the actual linear regression output depicting an adjusted value of 58% and a low p value 
indicating a statistically significant result 

• four customary residual plots confirming proper normality and lack of ordered patterns in the 
residuals associated with the regression model. 

The data shows that a statistically significant but moderate linear relationship exists between the 
utilization prediction from the AO BBN compared to the result of the validation survey scenario. 
A resulting “validation” linear regression model developed from this comparison produced the 
following equation: 

Validation = -357 + 11.1 BBN 

This equation relates the BBN model predictions to the outcomes of the validation interviews of 
scenarios as follows: Multiply the BBN prediction of the probability of using the WEA system for 
a given scenario by 11.1, then subtract 357 to arrive at the expected probability ascertained via the 
validation interview. As discussed earlier, the validation equation that relates the BBN model to 
the results of the validation interviews attempts to account for interaction effects of factors not 
realized in the model as well as any bias of the validation interviews. Consequently, additional 
validation interviews could confirm this relationship or modify it based on a broader set of AO 
inputs. 

We used the same process to validate the public tmst model. We created scenarios to exercise 35 
tmst factors of the model. We gave each factor a binary attribute; for example, the Action to be 
taken was either specified (+) or unspecified (-), and the Message confirmation from other 
sources was either available (+) or unavailable (-). Again, it was impractical to develop scenarios 
to address all combinations of tmst factor values; this would have resulted in 2^^ test cases. In¬ 
stead, we chose to categorize the factors in nine groups, as shown in Table 5. Within each catego¬ 
ry, we used fractional factorial methods to develop a series of test cases addressing the factors of 
the category. This resulted in the nine scenarios encompassing the 244 test cases shown in Table 
5. We show the scenarios in greater detail in Appendix U. 
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Table 5: Public Trust Model Validation Scenarios 


# 

Factors 

Cases 

1 

2 

3 

4 

5 

6 

7 

8 

1 

What has happened 

+ 

+ 

- 

- 


Who should act 

- 

+ 

+ 

- 

Why you should act 

+ 

- 

+ 

- 

2 

Action to take 

+ 

+ 

+ 

+ 

- 

- 

- 

- 

Time to act 

+ 

+ 

- 

- 

+ 

+ 

- 

- 

Who should act 

+ 

- 

- 

+ 

- 

+ 

+ 

- 

Lead time 

+ 

- 

+ 

- 

+ 

- 

+ 

- 

Relevance 

- 

+ 

+ 

- 

- 

+ 

+ 

- 

3 

Alert type 

+ 

+ 

+ 

+ 

- 

- 

- 

- 

Alert frequency 

- 

- 

+ 

+ 

- 

- 

+ 

+ 

Public outreach 

+ 

- 

+ 

- 

+ 

- 

+ 

- 

History or relevance 

+ 

- 

- 

+ 

- 

+ 

+ 

- 

4 

Relevance 

+ 

+ 

- 

- 


Clarity 

+ 

- 

+ 

- 

Confirmation 

- 

+ 

+ 

- 

5 

Coordination 

+ 

+ 

- 

- 

Confirmation 

+ 

- 

+ 

- 

Interpreted as spam 

- 

+ 

+ 

- 

6 

Coordination 

+ 

+ 

+ 

+ 

- 

- 

- 

- 

Clarity 

+ 

+ 

- 

- 

+ 

+ 

- 

- 

Language 

+ 

- 

+ 

- 

+ 

- 

+ 

- 

Multiple communication channels 

+ 

- 

- 

+ 

- 

+ 

+ 

- 

7 

What has happened 

+ 

+ 

+ 

+ 

- 

- 

- 

- 

Why you should act 

+ 

+ 

- 

- 

+ 

+ 

- 

- 

Relevance 

+ 

- 

+ 

- 

+ 

- 

+ 

- 

Clarity 

+ 

- 

- 

+ 

- 

+ 

+ 

- 

Confirmed in social media 

- 

+ 

+ 

- 

- 

+ 

+ 

- 

8 

Confirmation 

+ 

+ 

+ 

+ 

- 

- 

- 

- 

History of relevance 

+ 

+ 

- 

- 

+ 

+ 

- 

- 

Coordination 

+ 

- 

+ 

- 

+ 

- 

+ 

- 

Public outreach 

+ 

- 

- 

+ 

- 

+ 

+ 

- 

9 

‘All clear” message 

+ 

+ 

+ 

+ 

- 

- 

- 

- 

Alert source 

+ 

+ 

- 

- 

+ 

+ 

- 

- 

Alert frequency 

+ 

- 

+ 

- 

+ 

- 

+ 

- 

References 

+ 

- 

- 

+ 

- 

+ 

+ 

- 


Based on these test cases, we developed a questionnaire that solicited the respondents’ actions for 
each test case. We reproduce the questionnaire in Appendix V. 
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We empaneled a group of eight representatives of the public to validate the model. We chose rep¬ 
resentatives from the staff of the SEI who were not necessarily familiar with the WEA program or 
the WEA research being performed by SEE We presented the questionnaire to the panel and col¬ 
lected the individual responses. Appendix W summarizes the graphical results of these validation 
scenarios. We analyzed these results statistically to identify the mean, median, and variance of 
each response. 

We input these same scenarios into the BBN and compared the results with the panel responses. 
The public BBN validation employed a similar statistical analysis to the one conducted for the AO 
validation activity. We show the statistical analysis in Appendix X. Again, the linear regression 
analysis conducted depicts how we converted the probability predictions of different outcomes in 
the public BBN to align with the results of the public validation scenario results. 

Specifically, for each outcome in the public BBN (Relevance, Acting, ViewSpam, OptOut, Un¬ 
derstand, and Believe), we present 

• a table of the paired data from the BBN prediction and the results of the validation survey 
scenario 

• the scatterplot of the same data 

• the statistical linear regression output 

• the customary residual analysis associated with each regression equation 

In summary, we derived the following statistically significant equations to convert the BBN mod¬ 
el probability prediction to the probability assessment from the validation scenarios: 

Acting-V = -6.04 + 11.6 Acting-B 
Believe-V = -6.35 + 12.7 Believe-B 
ViewSpam-V = -0.407 + 2.09 ViewSpam-B 

The attempted linear regression associations of the Relevance, Opt-Out, and Understand factors 
did not produce significant statistical results. Several reasons again include the sample size of the 
data, the lack of appropriate modeling of factor interactions leading up to these outcomes, and the 
bias of the validation interviews conducted for this project for these specific outcome factors. As a 
result, more validation scenarios involving factors that influence Relevance, Opt-Out, and Under¬ 
stand factors would be helpful. These characteristics of the study could very likely produce varia¬ 
tion prohibiting a statistical relationship between the BBN and the validation scenario outcomes. 
Nevertheless, we could reasonably relate the relationship between the probability of Acting on a 
WEA alert to the validation interview results for the set of validation scenarios. 
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14 Conclusions and Future Work 


This task uncovered a rich legacy of research into the public’s response to emergency notification 
systems in the literature search. As a result, the focus of this task centered on the measurement 
and modeling of a varied set of factors thought to impact trust in the WEA system. The surveys 
used in this project appeared to be a viable mechanism to gauge the public’s reaction to different 
scenarios by describing scenarios in realistic fashion and asking members of the public and AO 
community to assess their likely responses. Testing the surveys ahead of time convinced us to 
move away from abstract, hypothetical questions to the evaluation of more realistically defined 
scenarios representing potential WEA alert situations. The probabilistic modeling appeared to 
provide the requisite ability to quantify the uncertainty in expert judgment and potential cause- 
effect relationships among a myriad of factors influencing bust in the WEA system. The greatest 
challenge, however, in the survey and modeling approach was the ability to confidently cover the 
space of interactions of different factors affecting bust. Although fractional factorial design of the 
validation cases enabled a sense of coverage during the validation phase, both sample-size limita¬ 
tions and the inability to reflect complex scenarios in a survey instmment presented challenges to 
the modeling effort. 

The approach of interviewing emergency notification experts to explore potential causal factors of 
tmst followed by the distribution of surveys to help confirm causal factors helped conbihute to the 
knowledge of factors affecting trust in the WEA system and provided a framework for follow-on 
research. The intent of the modeling approach in this project remained twofold: (1) to provide 
short-term feedback on factors affecting tmst in the WEA system and (2) to provide a framework 
to leverage ongoing research into tmst factors, such that the growing body of knowledge of tmst 
may be operationalized and shared. 

As discussed earlier, this model could easily incorporate ongoing research into factors affecting 
tmst in the WEA system and provide richer understanding and prediction of tmst scenarios. Spe¬ 
cifically, observational research could serve to confirm relationships that we discovered in this 
project and, more importantly, to help the emergency management field understand more complex 
scenarios of combined factors driving tmst. Additionally, this modeling could be readily expand¬ 
ed to accommodate specific segments of the population, geographic areas of the country, and the 
host of factors purposely excluded from this project, namely, the set of factors deemed uncontrol¬ 
lable to participants and stakeholders of the WEA system. We hope that the modeling approach in 
this project will motivate future researchers to use such probabilistic modeling to operationalize 
causal relationships in a fashion enabling stakeholders and members of the public to both under¬ 
stand and believe the tmst model results. 

The probabilistic tmst models in this report may be readily updated and used within future re¬ 
search in several simple ways. First, various forms of cause-effect modeling may be used with 
experts in the future to revisit the relationship of causal factors to the outcome of trust in the WEA 
system. Such cause-effect modeling may include cause-effect matrices as used in this project, 
Ishakawa (“fishbone”) diagrams, reconstmction of actual tmst situations, additional interviewing 
and observation of emergency notification staff and members of the public, and review of on¬ 
going research literature. Second, but much more challenging, future research could use con- 
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trolled experiments that expose live subjects to emergency notification scenarios and record and 
analyze their behavior. As seemingly prohibitive as this might sound, such research could employ 
methods not unlike the monitoring of household viewing experiences for the Neilson television 
ratings. 

In general, researchers can design BBN probabilistic models to learn over time as streams of data 
from real life are fed into the models. Both the “prior” distributions and “likelihood” or “joint 
likelihood” distributions may be revisited based on ongoing experiences and observations. Simi¬ 
larly to the learning mechanisms within email spam filters, these BBNs could learn from either 
continuous streams of data or batches of data over time. The primary challenge would be to define 
the desired learning process and secure the commitment of responsible organizations to provide 
the ongoing recorded data from emergency notification scenarios. 

As noted earlier, the primary limitations of this project’s modeling approach to bust in the WEA 
system rests in the challenge to analyze the truly expected response to the WEA system, whether 
it be an AO contemplating initiating a WEA alert or a member of the public responding to such an 
alert. The key question remains: How differently does actual behavior vary from the response to a 
scenario within a survey or interview? Research indicates that humans overestimate positive or 
desirable outcomes and underestimate negative or unwanted outcomes. Consequently, the survey 
approach of hypothetical scenarios may underestimate the effects of some of the negative trust 
factors and overestimate the effects of some of the positive influences on trust. Future research 
will need to be cognizant of this and investigate the human response accordingly. 
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Appendix A Human-Subject Research Application and 
Approval 

Human-Subject Research Application 


Carnegie Mellon University 


APPLICATION FOR IRB REVIEW OF RESEARCH INVOLVING HUMAN SUBJECTS 

(Not for exempt research) 

Please complete this application as thoroughly as possible. Your application should include the following: 

1. A consent form using the current CMU template that the participants and/or parent/guardian will be required to 
sign. 

2. A copy of any questionnaires, surveys, images, de briefings that will be used. 

3. A copy of any recruitment documents (including advertisements, flyers, letters, invitations, email) to be used; 

4. A copy of the training certificates for all individuals working on the research unless they are on file with the CMU 
IRB. Training is available at: http://www.citiprogram.org . See the IRB website for details. 

5. If the Pi is a student, the faculty advisor must submit a Faculty Advisor Assurance Form. 

Please email all documents to irb-review@and rew.cmu.ed u . For as^stance call the CMU Office of Research Integrity and 
Compliance @ 412-268-5460 or email irb-review@andrew.cmu.edu . Additional information and templates are available at 
http://www.cmu.edu/osp/regulatofv-compliance/human-subiects.html 


For IRB OffkcUso 

IRB No:_ 

Roc'd:_ 


1. Protocol 

Title: Wireless Emergency Alert System Trust Model 
I I This is a previously approved study that has lapsed. I Previous IRB No: H$ 


2. Principal Investigator (PI) 


Name: Robert W. Stoddard 11 

Department: SEI / SCPM 

Telephone: 412-268-1121 

1 E-mail: rws<®sei.cmu.edu 

Training Cert. || Attached ^ On File 

1 11 am a student. If so, please provide information about your faculty advisor below. 

Faculty Advisor Name: 

1 E-mail: 

Training Cert. || Attached LJ On File 


// a student is the PI. the faculty advisor must complete and submit a Faculty Advisor Assurance Form. 


If there is someone other than PI to correspond with regarding this protocol, please list below. 


Contact Person Name: 

1 Telephone: 

E-mail: 

Business Manager for your department: 

E-mail: 


3. Co-investigators 


Name: 

E-mail 


Training Cert. || Attached || On File 

Name: 

E-mail 


Training Cert. |\ Attached (] On File 

Name: 

E-mail 


Training Cert. || Attached || On File 

Name: 

E-mail 


Training Cert. CD Attached || On File 

Name: 

E-mail 


Training Cert. || Attached || On File 

Name: 

E-mail 


Training Cert. || Attached || On Flie 

Name: 

E-mail 


Training Cert. || Attached || On File 


4. Funding _ _ _ _ _ 

I I Unfunded research | [xl External Funding | I I Internal Funding 

Sponsor/Source: I I NSF I INIH DO Other, please specify: US Department of Homeland Security 


Grant Title: PWS 5-308J for contract FA8721-05-C-0003 between 5EI and DHS 
If federally funded, is this application consistent with the grant? [XI Yes I I ^ 

Is CMU the prime recipient of funding? ^ Yes Q I Is CMU a sub-recipient? | | Yes ^ No 


1 
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Carnegie Mellon University 


ForIRB OfficaUsa 


□ SPEX Proposal»: 


□ SPEX Award It: 


if you don't know the funding/grant information, please get it from your department's business manager. 


5. Protocol Description 


Provide, in lay terms, a summary of your proposed study as outlined below. You may attach the protocol to this form If you like. 
This research Includes a study aixi probabilistic model of the factors which may influence both the public trust and the alert 
originator trust in the new United States Wireless Emergency Alert (WEA) system, formerly called the Commercial Mobile Alert 
System (CMAS). Our team was asked to leverage existing research and collect any new additional data that would enable such a 
quantitative model of trust that would, in turn, be used to develop heuristics guiding the alert originators and ownersof the WEA 
system in what conditions and actions to be aware of that may degrade trust in WEA. This remains an important research activfCy 
to help ensure a successful deployment of WEA in the United States during the summer of 2013. Our specific research activity 
involving HSR lies in the planned conduct of online surveys to approximately 5000 members of the public and approximately 800 
members of the alert origination community across the country. 


b. What is the purpose of the study (what is your research question) and how will the data collected be used? The 
survey questions will enable our research team to quantify the probabilistic relationships in a Bayesian Belief 
Network (BBN) trust model that we have developed to predict the likelihood an alert is heard, understood, 
believed and acted upon. 


Describe the research procedures (include the activity(s), location(s) and time required of the participant).Online 
surveys will be issued during March to the employees of the Software Engineering Institute (~600), members of a 
local engineering society (~60) and members of the American Society for Quality Software and Reliability divisions 
(~4300). The respondents will be given approximately three weeks to complete the online surveys. The online 
surveys will be taken by participants at any internet connection of their choice. Each survey is expected to take no 
more than 5-8 minutes. 


d. Who will be asked to participate?SEI employees, local engineering society members and members from the 
American Society for Quality 


Will questionnaires or surveys be used? ^ Yes Q No If yes, please attach. 


f. Will tasks be done on a computer? ^ Yes Q No 

If yes, how will the tasks be accessed? ^ Remotely via the internet? Q In the research lab? Q Other, please 
explain; 


g. Will deception be used? Q Yes ^ No 

If yes, describe how participants will be debriefed. Please include the de-briefing material and/or script. 


h. In what country will the research be conducted (check all that apply)? ^ United States Q Qatar 
I I Other country, please list: 


Will the research be conducted on a CMU campus? ^ Pittsburgh Q Silicon Valley Q Qatar ^ No 

If no, please indicate the location(s). Non-CMU participants will be taking this online survey through any internet 

connection of their choice using a link provided by the Qualtrics invitation email. 

// applicable, please attach documentation of permission to conduct research in private, non-CMU space. 


6. Participants 


a. Will any of the following classes of vulnerable subjects be involved in the proposed study? (check all that apply) 


Class 


Pregnant women, human fetuses | | Yes | | No [XI Pregnant women will not be 
specifically included or excluded. 4 S«c 

hnp hKs.flov/ohfT)/human;ub:ects/gu*danc«/4Scfr46.htm. research that is incidental to prejnancy and has 

no risk to the fetus can only indude pregnant women if ALL aspects of Subpart B are met.) 


Comments 


We will be using provided 
email lists with no 
knowledge of whether 
individuals are or are not 
pregnant. 
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Carnegie Mellon University 


ForIRB OfficaUsa 

IRB No:_ 

Rac'd:_ 

Neonates I I Yes IXI~No 
Prisoners I I Yes IXI No 

Children I I Yes [Xl No 

”b! Individuals with compromised mental status I I Yes Dx] No 

If yes, indicate how this will be determ irted. 

~c. Will the participants be capable of understanding the nature of the study and the consent process? M Yes I I No 

If no, explain. 

d. What is the age range of participants in the proposed study? 18-75 

e. How many participants are needed for the study? 5000 

How was that number determ ined?3 different public surveys with 200 respondents each and assume a 14% 
response rate (4200 total needed); additionally two alert originator surveys with 200 respondents each and 
assume a 50% response rate (800 total needed) 

f. What do you estimate the ratio of males to females to be? 50% 

Will this be reflective of the local population? ^ Yes Q No 

Will you target a certain population? Yes Q No Please explain The only targeting that we are doing is to have 
the public respond to the public survey and the alert originator community to respond to the alert originator survey. 

g. Do you anticipate that your participants will represent a cross-section of the population in the region where the 
study is being conducted? Q Yes No 

If yes, please describe and estimate the percentage that will be from minority groups. 

If no, please describe your study population and address why minority representation is not considered. Our study 
population consists of members of the public without regards to minority status and focuses on a narrow set of 
attributes that the WEA Service can influence via the behavior of the WEA Service providers. We note that the email 
lists we have accessed are clearly biased towards the software and reliability members of the American Society for 
Quality. This scope will be clearly documented in the final report. 

h. Please list inclusion and exclusion criteria. Only alert originators will be allowed to take the alert originator survey. 
Other inclusion criteria will be age (18 yrs of age or older) and currently using some form of a cellular phone. 

7. Participant Recruitment 

a. Describe how participant recruitment will be performed. Include how and by whom potential participants are 
introduced to the study. Recruitment will be by email only based on assembled email lists. 

Check all boxes below that apply. 

I I Flyers Where will they be posted? 

□ Radio, TV 

1X1 E-mail solicitation Indicate how the email addresses are obtained.loc Elm is securing permission from the 
Director's Office via Linda Northrop to send out the email invites for the public survey to SEI employees. Joe Elm 
also jast secured permission to send the survey email invites to about 60 local members of an engineering society. 
Robert Stoddard will secure written permission from the American Society for Quality to send out the survey invite 
emails to approx. 4200 ASQ members of the Reliability and Software divisions. 

I I Web-based solicitation. Specify sites: 

□ Participant Pool. Specify what pool: 

I I Other, please specify: 

b. Will participants undergo screening prior to their participation? If yes, please describe. 

Please attach any recruiting materials you plan to use and the text of e-mail or web-based solicitations you will use. 

8. Consent 
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Carnegie Mellon University 


ForIRB OfficaUsa 

IRB No:_ 

Rac'd:_ 

~a. Do you plan to use consent forms? [XI Yes I I No 

If no, you must complete the section 8e below on waiver of informed consent. 

If yes, describe how consent will be obtained and by whom. We will include an online consent form at the beginning 
of the survey in which the respondent will then be required to check a consent box to continue the survey. Record 
of the consent info will be captured and retained by the Qualtrics survey software tool. 

Will consent be obtained online? ^ Yes No 

If yes, you must request a waiver of written documentation below in section 8h. 

b. If participants are minors will assent forms be used? I ]^es ) I No If no, please explain. ]XI NA no minors 

c. Will the consent form be presented on paper or online? Q Paper Online 

d. Are you requesting to use a consent format that is different from the CMU model consent? Q Yes ^ No 
If yes, please explain. 

e. Are you requesting a waiver of informed consent? Q Yes No 

If yes, please explain how each of the elements listed apply to your request for a waiver: 

■ The research involves no more than minimal risk to the subjects; 

■ The waiver will not adversely affect the rights and welfare of the subjects; 

■ The research could not practicably be carried out without the waiver and ; 

■ Whenever appropriate, the subjects will be provided with additional pertinent information after participation. 
Explain; 

f. Is the waiver for all study participants? Q Yes Q No 
If no, to whom does the waiver apply? 

g. Is the waiver for all study procedures? Q Yes Q No 
If no, to what procedures does the waiver apply? 

"h! Are you requesting a waiver of written documentation (signed) of informed consent? 1X1 Yes I I No 

If yes, please indicate which one of the following applies; 

^ The only record linking the participant and the research will be the consent document and the principal risk to 
the participant harm would be from breach of confidentiality. 

^ 1 consider this a minimal risk study that involves no procedures for which written consent is r>ormally required 
outside of research. 

i. Explain how the study meets the criteria checked above. No one outside of the principal investigator will have access 
to the survey results in which personally identifiable information is recorded. The nature of the questions 
represents minimal risk to a respondent. 

9. Risks and Benefits (Note: payments to participants are not considered to be a benefit) 

~a. Will participante receive intangible benefit from the study? [XI Yes I I No 

b. Discuss the direct and indirect benefits to participants. They will receive some familiarity with the nature and 
content of alert messages that will eventually be issued to mobile devices based on geographic location. 

c. Discuss the risks to participants. One risk will be that the 17-20 questions may appear slightly similar and 
require concentration. A second risk will be "breach of confidentiality" from a standpoint of obtaining personal 
email addresses and the technical ability within the password controlled Qualtrics tools to map email 
addresses to individual responses. 

d. Discuss how any risks will be managed and/or minimized. We are limiting each survey to 17-20 questions that 
may be accomplished in about 5-8 minutes. We will not force respondents to answer individual questions so 
that they may skip one or more questions if needed. We will also control access to email addresses and 
individual responses by limiting access to the Qualtrics survey tool database by password. Content and results 
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will be saved in offline eleetronic media in a secured container so that the online Qualtrics content will he 
deleted to secure from any future inadvertant access. 

e. If deception is involved, please explain. 

f. Indicate the degree of physical or psychological risk you believe the research poses to human subjects (check which 
one applies). 

^ Minimal Risk: A risk is minimal where the probability and magnitude of harm or discomfort anticipated in the 
proposed research are not greater, in and of themselves, than those ordir^rily encountered in daily life of during the 
performance o routine physical or psychological examinations or tests. 

I I Greater than Minimal Risk: A risk is greater than minimal where the probability and magnitude of harm or 
discomfort anticipated in the proposed research are greater than those ordinarily encountered in daily life or during the 
performarKe of routine physical or psychological examinations or tests. 

g. Describe how the study fits in this risk level. The survey is short in time (5 mins) and length (17-20 questions) and 
does not seek personal sensitive information. 

10. Participant Compensation and Costs 

a. Are participants to be compensated for the study? |\ Yes 1X1 No 

if yes, what is the amount, type and source of funds? 

Amount: | Source: | Type (giftcard,cash): 

b. Will participants who are students be offered class credit? || Yes ^ No 

If yes, please indentify the class and instructor. 

c. Are other inducements planned to recruit participants? Yes |_| No 

If yes, please describe. We will offer all respondents an invitation-only wehinar in the near future to provide 
more interesting details of the WEA system and it's general operations and purpose. 

d. Are there any costs to participants? || Yes ^ No 

If yes, please explain. 

e. Will you compensate participants for injury resulting from participation? |_| Yes |_| No 1X1 NA 

If yes, please describe. 

11. Confidentiality and Data Security 

a. Will personal identifiers be collected? (X Yes |_| No 

If yes, list the personal identifiers to be collected. The Qualtrics tools will maintain a linkage of a person's email 
address with a given survey response. Although this linkage is of not interest to the research, it will enable the 
survey software to issue reminder emails and control who is responding to the survey. 

b. Will identifiers be translated to a code? [XI Yes \| No 

If no, indicate why. 

c. Will audio recordings be made? |1 Yes 1X1 No 

If yes, please describe. 

d. Will video recordings be made? Q Yes ^ No 

If yes, please describe. 

e. Is the information so sensitive that you will obtain a certificate of confidentiality from NIH? Q Yes ^ No 

f. Who will have access to data (surveys, questionnaires, recordings, interview records, etc.)? Only the principal 
investigator will have access to the Qualtrics survey results. Summarized results will be analyzed by the 
principal investigator for purposes of populating conditional probabilities in the BBN probabilistic model. 

g. Describe how you will protect participant confidentiality and secure research records (Will they be stored on a 

secure computer, locked cabinet, etc?). All paper records will be secured under lock and key in the department 
of the principal investigator. Electronic files of the survey results will be saved and archived to CD and stored 
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with the paper records. All other online Qualtrics software files and results will then be purged once the 
storage media and paper is secure. 

h. Describe your process for monitoring data to ensure that study goals are met. (Review of lab notebooks, frequency 
of meetings to review data, who will be present at the meetings, how recruitment and retention will be monitored, 
etc.) This work will convene and last during the next three months, after which all records will be securely stored. 

12. Conflict of Interest 

Do you or any individual who is associated with or responsible for the design, the conduct of or the reporting of this 
research have an economic or financial Interest in, or act as an officer or director for any outside entity whose interests 
could reasor^bly appear to be affected by this research project? Q Yes ^ No 

If yes, please provide detailed information to permit the IRB to determine if such involvement should be disclosed to 
potential research subjects. 

13. Cooperating Institutions 

a. Is this research being done in cooperation with any institutions, individuals or organizations not affiliated with CMU? 
13 Yes □ No 

If yes, please list and describe their role In this research. SRA is on subcontract to the SEl to assist as necessary. 
Their primary role was an early influence on the structure of the probabilistic model and the collection and 
submission of the 800 alert originator email addresses to use for the survey. 

~b. Have you received IRB approval from another IRB for this study? f I Yes OO No I I Pending 

If yes, please attach a copy of the IRB approval. 

c. If multiple institutions are involved In this study indicate who is responsible for oversight of the entire study. SEl 
If applicable, please provide the name(s) and address{es) of all officials authorizing to access human subjects in 
cooperating institutions not affiliated with CMU. 

Please attach documentation of approval. 


I Principal Invest^atoKs Assurance Statement for Using Human Subjects in Research 

I certify that the information provided in this IRB application is complete and accurate. 

I understand that as Principal Investigator, I have ultimate responsibility for the conduct of IRB approved studies, the 
ethical performance of protocols, the protection of the rights and welfare of human participants, and strict adherence to 
the studies protocol and any stipulations imposed by Carnegie Mellon University Institutional Review Board. 

I understand that it is my responsibility to ensure that the human participants' involvement as described in the funding 
proposal(s) is consistent in principle, to that contained in the IRB application. I will submit modifications and/or changes 
to the IRB as necessary. 

I agree to comply with all Carnegie Mellon University policies and procedures, as well as with all applicable federal, 
state, and local laws, regarding the protection of human participants in research, including, but not limited to: 

• Ensuring all investigators and key study personnel have completed human subjects training program; 

• Ensuring protocols are conducted by qualified personnel following the approved IRB application; 

• Implementing no changes in approved IRB applications or informed consent documents without prior IRB approval in 
accordance with CMU IRB policy (except in an emergency, if necessary to safeguard the well-being of a human 
participant, and will report to the IRB within 1 day of such change); 

• Obtaining the legally effective informed consent from human participants or their representative, using only the 
currently approved date-stamped informed consent documents, and providing a copy to the participant. 

• Ensuring that only IRB-approved investigators for this study obtain informed consent from potential subjects. 
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• Informing participants of any relevant new information regarding their participation in the research that becomes 
available. 

• Promptly reporting to the IRB any new information involving risks to research participants, including reporting to the 
IRB, Data Safety and Monitoring Boards, sponsors and appropriate federal agencies any adverse experiences and all 
unanticipated problems involving risks to human subjects or others that occur in the course of the research. 

• If unavailable to conduct research personally, as when on sabbatical leave or vacation, arrangements for another 
investigator to assume direct responsibility for studies will be made through modification requests to the IRB; 

• Promptly providing the IRB with any information requested relative to protocols; 

• Promptly and completely complying with IRB decisions to suspend or withdraw approval for projects; 

• Obtaining Continuing Review approval prior to the date the approval for a study expires (approval for the study will 
automatically expire); 

• Maintaining accurate and complete research records, including, but not limited to, all informed consent documents 
for 3 years from the date of study completion; 

• Informing the CMU IRB of all locations in which human participants will be recruited for protocols and being 
responsible for obtaining and maintaining current IRB approvals/letters of cooperation when applicable; 

• Complying with federal, state and local laws and regulations and sponsor terms and conditions; and 

• Complying with CMU policies on the responsible conduct of research. 


Robert W. Stoddard _ 1 March 2013_ 

Principal Investigator Name and Signature Date 

Note: If e-mailed from the Pi's CMU e-mail account a hand written signature is not needed. Please type in name and date. 
If the PI is a student, the faculty advisor must submit a Faculty Advisor AssurarKe Form. 


Please email all documents to irb-review(g)andrew.cmu.edu . 

Note: Links to the policies and Federal regulations for the protection of human research subjects {including the Code of Federal 
Regulations (.CF.R.j Title 45 CFR Part 46 and Title 21 C.F.R. parts 50 and 56) are available on the IRB web page 

( http://www.cmu.edu/provost/spon-res/conipliance/hs.htm V 
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CMU Internal Review Board Approval 


('iiriiejfie IMt'lloii L iiivt‘rsily 


Office of Research Integrity and Compliance (ORiC) 
Carnegie Mellon UntversiTy 


Institutional Review Board 


5000 Forbes Avenue 
Warner Hall. A~ Root 


Federalwide Assurance No: FWA00004206 
IRB Registration No: IRB00000603 


Pittsbirgh. Pennsylvania 15213-3890 
412 268.7166 
ift»-rev»ewffeandrew emu edu 


Certification of IRB Approval 


IRB Protocol Number: 
Title: 

lnvestigator(5); 

Department(s): 

Date: 


HS13-137 

Wireless Emergency Alert System Trust Model 
Robert Stoddard 
SEI/SEPM 
March 29 , 2013 


Carnegie Mellon University Institutional Review Board (IRB) reviewed the above referenced research 
protocol in accordance with 45 CFR 46 and CMU's Federalwide Assurance. The research protocol has 
been given APPROVAL by Enpedited Review on March 22,2013, as authorized by 45 CFR 46.110 (7) and 
21 CFR 56.110. This APPROVAL expires on March 21,2014, unless suspended or terminated earlier by 
action of the IRB. 

The IRB has granted a waiver of written documentation of informed consent for this study. 

All untoward or adverse events occurring in the course of the protocol must be reported to the IRB within 
three (3) working days. Any additional modifications to this research protocol or advertising materials 
pertaining to the study must be submitted for review and granted IRB approval prior to implementation. 
Please refer to the above-referenced protocol number in all correspondence. 

Federal regulations require that all records relating to this research protocol be maintained for at least 
three (3) years after completion of the research, and be accessible for inspection and copying by 
authorized representatives at reasonable times and in a reasonable manner. 

The Investigator(s) listed above in conducting this protocol agree(s) to follow the recommendations of 
the IRB and the Office of the Provost of any conditions to or changes in procedure subsequent to this 
review. In undertaking the execution of the protocol, the investigator(s) further agree(s) to abide by all 
CMU research policies including, but not limited to the policies on responsible conduct research and 
conflict of interest. 

The IRB maintains ongoing review of ali projects involving humans or human materials, and at continuing 
intervals, projects will require update until completion. At the end of the current approval, a continuing 
review form, current application/protocol and current consent form(s) must be submitted by the PI to 
the IRB summarizing progress on the protocol during that period. Please be advised that the continuing 
review form requests information pertaining to women and minorities; therefore, this information should 
be tracked with your participants' data. Note that submitting for continuing review in a timely manner 
is the responsibility of the PI. 

Please call the Office of Research Integrity and Compliance at 412-268-7166 if you have any questions 
regarding this certification. Thank you. 



David Danks, Ph.D., IRB, Chair 
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Appendix B Interviews of Emergency Aiert Notification 
Officiais 


Interview Questions 

Concept of Trust 

1. Do you have a means of assessing and/or predicting the confidence that the public will have 
in your alerts? If yes, please describe it. 

2. What influences your decision to use a particular alert system? Please provide a specific ex¬ 
ample. 

3. Have your methods of assessing the public’s confidence in your alerts evolved over the 
years? How have they changed and why? 

Alerting Experience 

4. What methods do you use to issue alerts? 

telephone to subscribers 

email to subscribers 

reverse 911 

Twitter 

Facebook 

web site 

HAS (Emergency Alert System) 
outreach to media 
other (please describe) 

5. Have you consulted with anyone regarding ways to make your alerting system more effec¬ 
tive? Who? 

6. Do you have to coordinate with other organizations to issue an alert? Are there any problems 
in coordination? 

7. Do you have any issues with the propagation of your alerts? 

8. What criteria do you use to decide to issue an alert? 

9. What type of alerts do you issue and how frequently? 

10. Do you use multiple channels (e.g., email, telephone, EAS) to issue alerts containing corrob¬ 
orating information? How do you decide what to send through each channel? 

11. How often do you have to revise or amend an alert (other than an “all clear” notification)? 
Are the updates always sent via the same channels as the initial alert? What guidelines or cri¬ 
teria do you use for updating alerts? 

12. Would you please describe your process for changing how you issue alerts? What factors do 
you consider when making a change to your alert process? 

Alert Effectiveness 

13. What feedback do you get from issuing alerts, including feedback from the public? 

14. Do you collect data on the responses to the alerts you issue? What data do you collect and 
how do you use it? 
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15. How do you determine the effectiveness of your alerts? 

16. What causes the public to ignore an alert? 

17. Do you receive acknowledgement receipts to alerts you issue? Are the acknowledgement 
receipts analyzed by you and have they influenced how you issue alerts? 

18. How do you inform and educate the public regarding your alert system? How has that educa¬ 
tion affected the public’s response to your alerts? 

19. What should be done to improve alert systems? What would be the perfect alert system? 

20. Regarding the content of an alert, what makes it effective? What makes it ineffective? 

[WEA] Experience 

21. How does [WEA] figure into your alert strategy? What do you see as benefits and detrac¬ 
tions of using [WEA]? 

22. Please describe any experience you’ve had with [WEA]? How has that worked? Was it ef¬ 
fective? 

Additional Questions 

23. Are there any factors we’ve missed that you believe influence the public’s trust in alerts? 

24. Are there any factors that influence your use of an alert system, either positive or negative, 
that we have not covered? 

25. Would you be willing to discuss this topic further with us? We would like to get more in- 
depth information about your experiences. 

Thank you very much for your time and for sharing your expertise with us. 


Summarized Notes from Interviews of Emergency Alert Notification Officials 

Trust Model Benchmarking Interview Themes 

• Interviews conducted 10/10/12-10/24/12 

• Participant key: 

- MG 

- KH 

- LF 

- DM 

- FS 

- AC 

- LL 

- JF 

- RS 

Overarching Compelling Themes: 

1. Public response to alerts is directly tied to the relevance of alerts - [WEA]’s current geotar¬ 
geting configuration makes it unlikely that AOs will disseminate relevant alerts to a majority 
of constituents in a given jurisdiction due to few incidents that have county-wide impact; 
more than one alert originator noted that the public is looking for more personal alerts and 
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more than one jurisdiction allows the public to customize their alerts [alert types, alerting 
time of day, location, etc.]. 

2. The public, in general, is not interested in subscribing to an alert service (reflected in low 
numbers); even with heavy promotion subscriber numbers are relatively low; AOs seem 
concerned that this lack of interest may translate into high opt-out rates, which could already 
be taking place due to NWS [National Weather Service] alerts. 

3. Regarding [WEA], nationwide public education and AO education is essential to inform the 
public and gain its trust, prevent opt out from [WEA], and increase participation of AOs in 
[WEA] and their understanding of how the system works; appropriate education is not taking 
place, we are already “losing the battle.” Further, it is unclear who should take ownership of 
this process but it appears that collectively system owners and stakeholders are falling short 
at every level. 

4. There does not appear to be consistency in how AOs are using different alerting tools. Some 
use reverse 911 as their primary tool for severe incidents, and SM is used for general aware¬ 
ness; others take the opposite approach. There is, however, consistency in that AOs think 
[WEA] is valuable in theory, but its current 90-character limit and county-level geotargeting 
capability limit its applicability and value right now. 

5. Concept of Trust 

Assessing public trust is inconsistent among AOs. Generally, however, AOs seem to rely 
on SM to some degree [this seems to offer somewhat more robust and positive data than 
more traditional means such as calls or emails], internal conversations, occasionally un¬ 
solicited feedback from the public via email, media commentary, and, rarely, formal 
studies or surveys. AOs do not appear to be recording or storing feedback or assessment 
data, except in the case of a formal study or survey. 

AOs did not indicate that the public’s trust or feedback factors into the use or adoption 
of new systems; however, the fact that all AOs we spoke to mention some use of SM, 
which is now ubiquitous in the public realm, may indicate some correlation between di¬ 
rect or indirect feedback from the public and the adoption of new alert tools. Direct or 
indirect feedback does appear to have some influence over how current tools are used, 
when some tools are used, and the types of alerts that are issued, namely in the case of 
less severe events. 

6. Alert Experience 

There is a lack of consistency in how AOs are using popular alerting tools. For example, 
some use reverse 911 as their primary tool for severe incidents, and SM [social media] is 
used for general awareness; others take the opposite approach. All AOs employ more 
than one tool to alert and/or communicate emergency information to the public. 

With the exception of NWS, AOs did not provide or indicate specific criteria for issuing 
or amending alerts beyond generalities such as “life safety impact” and “[event] will af¬ 
fect people.” Frequency of alerts varies widely and in many cases is dependent upon 
seasonal activity, such as wildfire or hurricane season. 

Not all AOs amend or cancel alerts using the same tool employed for issuing the original 
alert. A few AOs stated that amending or canceling alerts was rare. A few AOs noted the 
importance of issuing a final communication to the public to close out an alert event. 
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Overall, AOs did not report major concerns in the areas of coordination and propagation. 
It is worth noting, however, that due to the potential for message bleedover when using 
[WEA], one AO noted an expectation that there will be coordination challenges with 
neighboring jurisdictions. 

7. Alert Effectiveness 

Public response to alerts is directly tied to the relevance of alerts - [WEAj’s current geo¬ 
targeting configuration makes it unlikely that AOs will disseminate relevant alerts to a 
majority of constituents. In a given jurisdiction, AOs indicate there are a few incidents 
that have a county-wide impact that merit issuing a [WEA] alert. 

The public, in general, is not interested in subscribing to an alert service (reflected in 
low numbers). Heavy promotion and education of local alerting services and processes, 
while not the norm, still results in a relatively low proportion of the public subscribing. 
Further, AOs seem concerned that this lack of interest may translate into high opt-out 
rates, which could already be taking place due to the large number of NWS [WEA] 
alerts and general lack of education regarding alerting generally and [WEA] specifically. 
Again, AOs do not appear to be recording or storing feedback or assessment data, except 
in the case of a formal study or survey feedback, as they noted when questioned regard¬ 
ing the assessment of public trust, is gathered generally via SM, internal conversations, 
unsolicited feedback from the public via email, media commentary, and, rarely, formal 
studies or surveys. Public acknowledgment or tracking of “acks” from the public is rare 
due to liability concerns and/or technology or staffing limitations. 

8. [WEA] Experience 

AOs generally think [WEA] is valuable in theory, but its current 90-character limit and 
county-level geotargeting capability limit its applicability and value right now. AOs are 
taking a cautious approach to issuing [WEA] alerts and overwhelmingly believe the pub¬ 
lic has little to no knowledge about the service. 

AOs say nationwide public education and AO education is essential to inform the public 
and gain its trust in [WEA], prevent opt out from [WEA], and increase participation of 
AOs in [WEA] and level-set or enhance AOs’ understanding of how the system works; 
appropriate education is not taking place, we are already “losing the battle.” Further, it is 
unclear who should take ownership of this process but it appears that collectively system 
owners and stakeholders are falling short at every level. 

Most AOs’ experience, including those who have adopted [WEA], is cited as experience 
with NWS’s [WEA] weather alerts. Feedback here is inconsistent; some AOs worry that 
NWS is over-alerting and generally doesn’t coordinate alerts with the jurisdiction well, 
and others plan to apply NWS’s [WEA] lessons learned. 

Summaries by Question 

1. Do you have a means of assessing and/or predicting the confidence that the public will have 
in your alerts? If yes, please describe it. 

AOs conduct surveys, formal or informal, following an incident/alert event (KH, MG) 
Social media use is growing, important to keep viral messaging consistent somehow 
(MG) 
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Related note — it is difficult to control who gets their alerts due to “a lot of spillover” 
(KH) 

Rely heavily on the media to mirror the message and provide follow up/more detailed in¬ 
formation (KH) 

Public doesn’t understand how system works - disconnect between public’s use of alert 
system and officials’ (KH) 

AOs monitor feedback of mass notification system, road reports, media stories, as indi¬ 
cators of public taking the recommended alert action (FS) 

County-wide incidents are rare, [WEA] alerts are designed to be relevant to an entire 
county (FS) 

Methods don’t exist here... (AC) 

Outside of doing a survey via a (rare) third-party organization, we don’t assess for these 
types of alerts; we depend upon our feedback from third-party sources - newspaper cov¬ 
erage of large-scale events, how well did reverse 911 messages work, did public follow 
them, etc.; gather some social media (SM) feedback, our alerts are one way (LL) 

Really only have feedback from SM, including Twitter & Facebook, which makes it eas¬ 
ier for the public to express their opinions; prior to SM, really only heard from people 
who wanted to complain (JF) 

Don’t have format to test it; do have historical email responses from the last 10 years 
(RS) 

[We have] sent out a few Survey Monkey surveys over last few years; 3-5 question sur¬ 
veys, going only to alert service subscribers (RS) 

Prompts to send surveys: the Mineral, VA Earthquake/Hurricane Irene/one additional 
weather event prompted [us] to send [our] most recent survey; typically received posi¬ 
tive responses, information about technical issues, questions like “why didn’t you alert 
earlier?” or “message wasn’t clear to me” (RS) 

For feedback that highlights technical glitches and are a high-priority issue.. .we’ll fol¬ 
low up with customer first to determine if it is operator error; if there is a true issue with 
process or policy and we feel it really needs to change, we’ll change (RS) 

Email responses from subscribers [the public] can be sent to report issues; typically [we] 
get responses about big events, received comments on messaging around Sandy, the 
Derecho — these comments were more about technical glitches with phones, devices to 
which the public has subscribed to receive alerts, and not responses to the actual messag¬ 
ing from [our office] (RS) 

Have your methods of assessing the public’s confidence in your alerts evolved over the 
years? How have they changed and why? 

Continuing to survey (conduct outreach/town halls instead depending on event) (KH) 
Wordsmithing messages gone wrong in the middle of events (KH) 

More impact-oriented alerts; more methodical surveying, surveying more often, follow 
up on survey lessons learned more consistently, factoring in changes to alert dissemina¬ 
tion methodology (MG) 

Continuing to focus on effective ways to communicate information (MG) 
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AOs need to build trust well ahead of time prior to when we actually issue the alerts; di¬ 
rect engagement with the public is key to completing this activity (FS) 

We recently (five years ago) established public notification office tasked with directly 
communicating with the public; previously communications were routed through the 
media (FS) 

Communication is conducted through the alerting system, SM, and our website (FS) 
Methods don’t exist here (AC) 

We’ve looked at what works and didn’t; looking at new vendor (LL) 

Exploring new means/tools to communicate with public; looking into systems that can 
take our message and translate/convert into more languages (LL) 

Don’t have a way to assess public’s confidence in alerts; when conducting my thesis, I 
found very little confidence in the government at that time; anticipate this confidence 
will grow as we shift from a one-size-fits-all approach to using more alerting methods 
(JF) 

We have tried to increase transparency and get the community’s input on tasks we previ¬ 
ously conducted internally (JF) 

We typically do an after-action program on any exercise or big event for which we stand 
up the emergency operations center (EOC); the Derecho was so significant that the 
board of supervisors asked us to expand and reach out to the community; we sent a Sur¬ 
vey Monkey of 15 quesfions fo county’s resident distribution lists (e.g., home owners as¬ 
sociations); also surveyed via the board of supervisors and chambers of commerce’s 
distribution lists; focus of survey was on how message was received.. .received approx¬ 
imately 19,000 responses on questions about alerting throughout the whole 4-5 day 
event; responses included (RS): 

■ Concerns that messages didn’t go out soon enough (8.30pm warning for a 12AM 
event) 

o We don’t have 24-hour program in [our city], more timely information would 
require a 24-hour agency 

■ Alerted the public using the “whole shebang” - Twitter, press conference, text alerts, 
etc. 

■ Public complained they didn’t have power, didn’t get messages — many tools with 
SM, alert network, Channel 16 in county, highway advisory radio—also have a full 
system of 7 towers dedicated to emergency messaging (don’t advertise this service, 
little response/interest by the public) 

■ Some people just weren’t paying attention; had power and received SM messages; 
got messages continually because members of public were charging devices outside 
the home (RS) 

What influences your decision to use a particular alert system? Please provide a specific ex¬ 
ample. 

We establish tool for disseminating information and are reluctant to sway from it — want 
public to see consistency in how/which tool they receive messages about particular 
events (MG) 
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Focus on making our data readily accessible — multiple sources that can be tweeted, etc. 
(MG) 

We need control over exactly what events we alert about (e.g., EAS is predetermined by 
FCC); forecasters don’t have direct control over parameters we’d use for [WEA] (urgen¬ 
cy, severity, certainty) (MG) 

Put control of how messages are rendered in hands of the receiver (they decide which 
levels they want to be alerted of) (MG) 

Accept inconsistency in how alerts are disseminated (some at county level, some below) 
due to carrier inconsistency — usually issuing [WEA] alerts via polygon, not FIPS (MG) 
Concerned about irrelevant alerts reaching public, due to county level geotargeting (MG) 
Use [WEA] for short fuse events; also use EAS because it will reach more people during 
a short fuse event; longer fuse event — there is more time to get an alert out through other 
methods (MG) 

Focused on acquiring tools only if they are interoperable with current system, eliminate 
additional steps either manually or with policy — one-click use (KH) 

We use a tiered approach tied to how localized an event is: SM and website is for gen¬ 
eral awareness, then alert system, then EAS to “yell at everyone to listen” (FS) 

We house our own opt-in alerting system [own the database of users/subscribers] and 
only use vendor as gateway; we don’t want to be dependent on the vendor, want to own 
our data (AC) 

We want portability and options on the go (AC) 

[WEA] is an amazing tool, will allow us to reach all of our tourists (AC) 

Timing — if we do not have much time and the event is fast moving we may use EAS; if 
there is time to geotarget, we will send out through our alert service; if there is time to 
add additional languages, we’ll do that also (LL) 

There is no one service that reaches everybody; looking at redundant systems; we want 
to get to the point that I enter data into one mechanism and the alerts go out to all our 
systems (JF) 

[A government agency] did an RFI earlier this year to look at other systems out there, 
concluded some others may be better and provide capabilities we don’t currently 
have.. .it is a huge project, tied in with the [state], military and federal entities, [and re¬ 
gional authorities]; federal government also requires that power plants need a nuclear 
alert system; full requirements document has been created - 45 items — calls for GIS 
mapping, report out capability, reverse 911 feature, CAP 2.0 compliancy for [WEA], etc. 
(RS) 

Cost is a concern — UASI funding to end in the next couple years (RS) 

What type of alerts do you issue and how frequently? 

Alerts for large-scale incidents are rare (KH) 

Most alerts are high in urgency and severity in small geographic area (KH) 

Short fuse alerts — up to approximately 700 in a month (flash flood, blizzard warning, 
tornado warning) 

Looking to better understand how public responds to our alerts over time; will revise as 
needed (MG) 


CMU/SEI-2013-SR-021 | 58 


We are info-centric — want to ensure everyone gets the info; pretty stagnant regarding 
which tools are used for which events (MG) 

We alert for public- or life-safety consequences, imminent threats: “hurricanes sell here,” 
flooding, snowstorms, wildfires, hazmat, high security/national significance and major 
traffic disruptions (e.g.. Super Bowl) (FS) 

Promote the alert service with clickable icons and “follow us” requests in alerts and 
website communications (FS) 

We send alerts two times per month over average, but every other day during hurricane 
season (FS) 

Transit and major transportation disruptions (probably 50 a day), AMBER, civil emer¬ 
gencies, police incidents after crime; major transportation disruption; hurricane warn¬ 
ings; weather (not often); we would issue more hurricane warnings but there is a lot of 
buildup in the media already (AC) 

Alert system interfaces with the CAD system — automatically alerts agency officials of 
incidents relevant to them; we test this morning, afternoon, and evening to ensure auto¬ 
matic dissemination is working (AC) 

Missing child; evacuations (have issued a couple during this fire season); more likely for 
us to use it during fire season; jurisdictions use system for test exercise (LL) 
Predominantly weather alerts, which we have tried to automate — alert comes right from 
NWS to the public, we try to avoid being the middleman; we are the middleman for non¬ 
weather alerts (JF) 

Fairfax has 2 systems in 1 (RS) 

■ 1: Community Emergency Alert Network (CEAN) 

o Residents can sign up multiple devices, choose to receive a variety of alerts like 
traffic, terrorist, weather, etc. 

o Business can sign up to 5 people - e.g., [one retail] mall has signed up 5 securi¬ 
ty personnel 

■ 2: Emergency Alert Network (EAN) — service to alert county employees; mandate 
that every county employee has an account (approximately 14,000 employees) 

o Each agency in local government has its own administrative capability; each 
creates its own groups so it can send messaging to its group 
o Fire and rescue uses it daily to alert chiefs of big fires 24 hours a day (RS) 

Police Department’s public affairs office can just alert a particular neighborhood of a 
home invasion; [one area that] typically floods — has a special River Watch group - used 
quite a bit last week; use the systems every single day; today sent traffic alerts; more 
than 100 messages go out, but a lot just to government employees; today, alerted about a 
big accident on [the] Parkway (RS) 

Weather another big one — last week sent thousands of messages, EOC opening/closing, 
requests for staffing; during 2009 Inauguration, we sent out 3.5 million pages - used a 
lot (RS) 

What methods do you use to issue alerts? Flow do you decide what to send through each 
channel? 

telephone to subscribers 
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■ KH: see reverse 911 (all subscribed or unsubscribed get calls) 

■ LL: cell phone calls to subscribers (couples with reverse 911 service); smartphones 
have closed-caption messaging being added; video, too 

■ RS: yes, as part of BAN & CEAN system (subscribers can list multiple numbers — 
including pager numbers) 

email to subscribers 

■ MG: yes to public officials (used less often now) 

■ FS: yes 

■ AC: Email subscribers get most info — this is most common alert medium from mi¬ 
nor to severe 

■ JF: alert via email but not to subscribers - emails only go to key POCs in the county 
(with the hope that they will relay them to their particular constituents.. .sometimes 
it works, sometimes it doesn’t) 

■ RS: yes, as part of BAN & CEAN system 

reverse 911 

■ MG: no 

■ KH: yes (people less sensitive to calls now) 

■ LL: our primary/true alert system, residents automatically a part of this system 

Twitter 

■ MG: still figuring out how to use SM 

■ KH: virtually all incidents/low interruption tool 

■ FS: once we issue an alert, almost instantly posted to Twitter 

■ AC: social media is used for more significant, severe events; If an emergency is big 
enough, more severe, we will issue alerts through Twitter, Facebook, our website 

■ LL: yes 

■ JF: yes 

■ RS: Yes. If event grows, may put on blog, SM, etc.; use this quite a bit during events 
- notify about shelters, preparation, issue updates throughout day (along with Face- 
book, blog) 

Facebook 

■ MG: still figuring out how to use SM 

■ KH: virtually all incidents/low interruption tool 

■ FS: once we issue an alert, almost instantly posted to Twitter 

■ AC: social media is used for more significant, severe events; If an emergency is big 
enough, more severe, we will issue alerts through Twitter, Facebook, our website 

■ LL: yes 

■ JF: yes 

■ RS: If event grows, may put on blog, SM, etc.; use this quite a bit during events - 
notify about shelters, preparation, issue updates throughout day (along with Twitter, 
blog) 

Website 

■ MG: yes 
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■ KH: virtually all incidents/low interruption tool 

■ FS: pools regional joint information center information 

■ AC: if an emergency is big enough, more severe, we will issue alerts through Twit¬ 
ter, Facebook, our website 

■ LL: yes 

■ JF: yes 

■ RS: [blog] 

HAS 

■ MG: yes 

■ KH: reserved for most urgent/severe 

■ AC: EAS is issued very rarely — haven’t issued an EAS message in the last 2.5 years 
- we rely on NWS for this purpose 

■ JF: probably our primary tool along with media; can do local EAS interraptions, but 
not everyone listens to local radio stations 

■ *RS: NWS - if there is something big in [our city] or whole region, use NOAA 
Weather Radio — can ask them to push out an alert message 

outreach to media 

■ KH: yes 

■ MG: via EAS to broadcasters/private meteorologists; KH: virtually all incidents/low 
interruption tool 

■ LL: yes 

■ JF: probably our primary tool along with EAS 

■ RS: good relationship with the media; joint information center with 7 staffers stood 
up when EOC is stood up 

Other (please describe) 

■ MG: communicate via satellite/landline to government officials, state and local EMs 
to disseminate info 

■ Sirens 

o KH: reserved for most urgent/severe (highly disturbing) 

o JF: have a countywide siren network but cannot geotarget it, would take signifi¬ 
cant money to upgrade 

■ [WEA] 

o KH: reserved for most urgent/severe (lack of geo-t, highly disturbing) 

■ Weather radio (technically this is part of EAS) 

o KH: reserved for most urgent/severe (lack of geo-t, highly disturbing) 
o JF: “Alert FM” through FM radio (As alert FM grows, we can geotarget better — 
but still that’s hit or miss, if you don’t have an FM Alert unit, you don’t get the 
message) 

o *RS: NWS — if there is something big in [our city] or whole region use NOAA 
Weather Radio — can ask them to push out an alert message 

■ SMS Text 

o KH: reserved for most urgent/severe (lack of geo-t, highly disturbing) 
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o FS: reserve for only imminent threats 

■ Other Radio 

o RS: XM Radio partnership — LED screen crawl message on the local XM radio 
channel 

■ Blog 

o RS: Yes. If event grows, may put on blog, SM, etc.; use this quite a bit during 
events - notify about shelters, preparation, issue updates throughout day (along 
with Facebook, Twitter) 

■ DOT 

o RS: Public affairs folks record message on hard radio - [the] DOT will post on 
electronic highway signs “switch to channel x to hear an important message” 

■ TV Crawl 

o RS: TV Channel 16 

Additional notes: 

MG: we are consistent in tools we use 

KH: we have found public expects us to alert through certain tools - e.g., phone calls 
FS: alert via a regional interoperability office, use a template that includes content of 
message and where to point the public (e.g.. National Weather Service, Hazmat hotline) 
for additional information because we don’t own 90% of the information shared during 
an event (e.g., weather, traffic, schools, utilities information/status) 

FS: we have two lists of subscribers and we send alerts to one or the other, or both, de¬ 
pending on the incident: public, media 

AC: different thresholds for alerting, really just is intuitive of what we think will benefit 
the public; also depends on time of day of an incident: if it’s 2 AM and less serious 
threat, send an email to minimize disruption, but probably follow up with a voicemail 
alert later in the morning 

AC: Subscribers sign up based on zip code and are also placed automatically into a par¬ 
ticular city; subscribers are able to choose the alert type that they want — transit, emer¬ 
gency, AMBER, etc.; we are also working on a time window, so if they only want Tam 
to 7pm to be alerted, that’s the only time they’d receive alert messages 
AC: we don’t like to go above 157 characters and we try to use text only when necessary 
LL: right now languages are just through voice, text, email, video, closed captioning, 
sign language, and Braille readers; looking to expand languages 
LL: In alert, direct public to call 211 to get more information, to find shelters, etc. 

JF: We have many gaps; reaching transients/visitors is our biggest issue 
JF: Don’t do reverse 911, landlines going by the wayside 

JF: I’d really like to streamline our system while maintaining its integrity, afraid too 
many people will have access to these alerts and warnings dissemination points; we’ll 
have an event and they’ll use it incorrectly, turn off the public, and then they stop want¬ 
ing the alerts, turning off the devices (e.g., turning off their radio); trust really has to be 
there, has to be well controlled 

JF: My jurisdiction is developing a citizens’ mobile app that is two-way — app users can 
get NWS warnings from us but they’ll also have access to mapping — they can see 
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if7where we open shelter; plus, if they see strange weather formations, for example, they 
can send it to us, and it will automatically be mapped within seconds for our responders; 
we will do it on an unofficial basis to start; we’ve done virtual cities — watch storms 
come in radar-wise and then see 911 calls pop up in that area; with citizen reports, we’ll 
know where to deploy resources, it will help us to prioritize 

■ Catalyst: need to be able to obtain immediate information from the public to create 
an overall picture - we are using a grant, working to build the app with SPAWAR & 
SAIC 

RS: BAN & CEAN are go-to on daily basis - weather, traffic, basic (communicates via 
pager, email, cell phones, etc. - multiple devices can be listed by subscriber) 

RS: For a weather watch — we send via email; for a weather warning — send via email 
and pagers, phones, etc. 

RS: The Region obtained UASI grant funds around 2001 — procured a system called 
Roam Secure Alert Network; all jurisdictions in [the region] have capability to send 
alerts using this program; system is very functional, each locality can test, can customize 
the system to meet their needs; only 75,000 subscribers 

RS: “Geofeedia” - geo-based system we use if we have a special event/area - e.g., can 
highlight a whole campus and show us every single person sending tweets, using Insta- 
gram, etc. if their phones are enabled for location capability — we can see their pictures, 
tweets and their x and y coordinates; we even will interface with folks — message with 
those in the map 

RS: If there is an HOC activation, we use everything to alert the public, will use 
YouTube to post quick video updates; stand up a joint information center with staff of 7 
fo communicate with media 

Do you have to coordinate with other organizations to issue an alert? Are there any problems 
in coordination? 

We alert on behalf of other organizations like police or fire (KH) 

No real coordination issues — we train them and then they call us when they need to alert 
(KH) 

Sometimes need to coordinate with multiple EMs in a single area (MG) 

Conduct a lot of pre-event coordination — outreach with EMs, the public, broadcasters, 
SKYWam training sessions for volunteers (MG) 

Real-time chat with EMs, government officials, broadcasters to obtain ground tmth 
(MG) 

Coordinate with other groups like Army Corps of Engineers etc. depending on type of 
alert to be issued (MG) 

Challenge: very small local staffs to coordinate with all these groups (MG) 

Usually no, but for weather-related incidents we have a great relationship with NWS, 
they supplement what we do; regarding [WEA] our agency has a clear understanding of 
what triggers an NWS [WEA] alert and the language they use; in future, when imminent 
threat weather event doesn’t meet NWS trigger criteria, we have agreed that we will is¬ 
sue the weather alert instead (FS) 
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Some regional joint information center coordination is required; it is important to be 
transparent with a common operating picture because it all ties to a larger regional re¬ 
sponse (FS) 

Some agencies in the jurisdiction have their own public information authorities; there is 
a lot of coordination if some of these particular agencies are responding to an incident— 
in which case we service as “trusted disseminator” on their behalf (FS) 

We operate according to the “three Cs”: coordination, collaboration, and communication 
drives the model of how we do business on emergency management side of things; we 
have yet to read an after action report that doesn’t prescribe improving communication 
(FS) 

We are moving from a model of need to know to need to share (FS) 

Technically we have capability to issue without consulting any other organization; don’t 
want to duplicate weather alerts, so we first see what NWS is doing before we issue a 
message, generally leave weather to NWS (AC) 

No coordination issues; jurisdictions in county able to issue alerts themselves unless 
they need help from us; if we evacuate we coordinate with sheriff’s department, but this 
doesn’t require an approval to actually issue the alert (LL) 

NWS sends their own weather alerts here, but don’t tell us they’ll do it — that is one issue 
we have! (LL) 

For local alerts, it’s just us sending the alert; we had a local group also alerting.. .they 
had access code and put out an inconsistent message and it confused people; if it’s an in¬ 
ternational alert, we go back and forth to make sure everyone is consistent in messaging 
(JF) 

The more we can fine-tune our alerts geographically, the better (JF) 

If it’s a generic message to all county residents and is not for a life-threatening event, 
we’ll coordinate with public affairs on wording; for something like a tornado; we don’t 
coordinate, just send the message out ASAP (RS) 

For the Northeast Shake Rattle & Roll Event - coordinated messages with the office of 
public affairs 2 weeks before sending them; we have weather templates, county & school 
closing templates — just need to insert date and time and then send the message (RS) 
NWS just started a new program - 3 meteorologists work directly with local emergency 
managers; we [regional authorities, sometimes the state too] will have a conference call 
with NWS and say “what’s the weather look like?” and then we push out alerts as need¬ 
ed; the NWS group is being proactive - they now will message the NCS with the best & 
worst case scenarios and what is actually expected - all provided on 1 slide — NWS 
started this only within last couple months (RS) 

We don’t coordinate messages with NWS, haven’t sent conflicting messages yet; Virgin¬ 
ia DOT has an on-site meteorologist (RS) 

Police department has access to send out alerts for law enforcement events because they 
need to alert the general public (RS) 

What criteria do you use to decide to issue an alert? 

We determined nine (9) sets of criteria for issuing [WEA] alerts (MG) 


CMU/SEI-2013-SR-021 | 64 


We have a specific set of criteria for each event type and alerting tool — we try to be very 
consistent in how we alert so public knows what to expect (MG) 

Only criteria: imminent threat to life and safety; we do not publicize a particular list of 
events/tools that we’ll use — too limiting, will cause us to freeze if something 
new/unexpected occurs and doesn’t align with our protocols (KH) 

Public safety or life safety impact; what is inconvenient for 80% of county could be life¬ 
saving for other 20% - and in that case, we will issue an alert (FS) 

We must trast agencies and local jurisdictions on whose behalf we issue alerts; because 
they think something is lifesaving, we take it seriously and issue the alert (FS) 

Our threshold is dependent on how many people it will affect; if we feel it will effect so 
many people, then it’s necessary to email; different agencies have different criteria, for 
example transit issues alerts for every little thing — bus detour, escalator outage, etc. 

(AC) 

Cannot provide specific criteria, it’s simply incident specific; if we can put information 
out though SM and update website if there’s fire in your area, then we will (LL) 

Try to use alert system for high level emergencies, e.g., evacuation or shelter in place ac¬ 
tion (LL) 

Life-threatening, urgency; we wouldn’t activate our system for something less severe 
(e.g., boil water) (JF) 

Traffic — subscriber determined; Terrorist — send to all users; Policies in place based on 
weather events (RS): 

■ for watch only will send to a group within the system signed up to receive those 
alerts types of alerts, and within the time frame dictated by subscriber (e.g., 9-5, 24 
hours) if warning, like severe thunder — 

■ for a tornado — send to all users across county 

■ Traffic - subscriber determined 

■ Terrorist — send to all users (RS) 

Flow often do you have to revise or amend an alert (other than an “all clear” notification)? 
Are the updates always sent via the same channels as the initial alert? What guidelines or cri¬ 
teria do you use for updating alerts? 

Infrequent changes - if protective action changes, impact area changes, or there’s an es¬ 
calation, we’ll revise alert, may add additional tools to issue the revised alert (KH) 
Update content, cancel, expire, expansion of warning area, new information that is sig¬ 
nificant and important to people affected; it’s important to reiterate that many times; 
most times — NWS doesn’t use the exact same channel or method to disseminate the up¬ 
dates as was applied to alert (MG) 

Cannot cancel [WEA] alerts (may cancel alert in all areas instead of the limited area that 
needs to be canceled), block updates from going through [WEA] (90 characters doesn’t 
allow for it) (MG) 

Depends on incident; if we alert about a storm that doesn’t materialize, we send out mes¬ 
sages in consumer-friendly terms — here’s what’s happening, here’s a link if you want 
more information (FS) 
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Need to keep coastal weather threats on people’s radar with steady flow of messages un¬ 
til official close-out message at end of event (FS) 

We must be particularly careful for unfamiliar/uncommon incidents, in such cases we is¬ 
sue a more steady flow of messages, ongoing alerting effort, exhaustive messaging to 
keep public engaged (explanation of situation, updates, actions, latest outlook, etc.) (FS) 
Updates always sent on same channels as the initial alert (FS) 

Rarely - don’t recall ever amending an alert; if a revision/amendment is sent, it would be 
sent via same channel used for initial alert (AC) 

Most of our updates are related to evacuation notices; only amendment would be adding 
to area of evacuation; send updates via the same channel as initial alert (LL) 

Regarding weather, we don’t issue “all clears” (JF) 

Normally the message goes out just once, unless the event changes, is extended, grows; 
we’ll likely tell the media to relay additional information and are less likely to go 
through all the alert channels, especially. If it’s just an inconvenience event, rather than 
an imminent threat (JF) 

No written criteria for our agency; especially on the international side along the water; 
every situation is unique, wind direction, etc. can all have an effect — we make alert de¬ 
cisions based on each event (JF) 

We send updates via the same channels; only “all clear” is if we have a police event in a 
specific neighborhood - that’s just once every few months (RS) 

Weather — never unless forecast changes; for a severe traffic event — send out 3 — the ini¬ 
tial, the update approximately 15 minutes later if a 1—2 hour closure, and the resolution; 
public does say we send too many messages, but we don’t care - you’d can’t be perfect 
with this (RS) 

Do you have any issues with the propagation of your alerts? 

Geotargeting is the #1 issue - causes tremendous overreach (MG) 

Tool’s ability to geotarget plays into our use of it; cell phones, emails, social media all 
disseminate quickly; we can only geotarget with landlines and sirens.. .but landlines are 
weakest link in system (KH) 

No, we have built trust with the public; we also recognize civic groups, etc. can carry 
more weight than we can; we tell our partners, liaisons, nonprofits to feel free to re¬ 
disseminate our information to their constituents; their propagation goes further than we 
could do ourselves (FS) 

I’d send more alerts, but it’s up to the bosses, who aren’t in favor of sending such [storm 
watch] alerts; I don’t know why more alerts aren’t sent (AC) 

I don’t think politics is an issue down here; when I worked in NYC — absolutely had is¬ 
sues; couldn’t believe what we would alert for; we had 5 staffers dedicated to sending 
alerts -sometimes we just seemed to alert as a way of demonstrating they were earning 
their salaries (AC) 

No propagation issues (LL) 

No propagation issues; It’s frustrating that it takes a period of time to alert via 6 different 
mediums (JF) 
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7-8 years ago there were many issues but they are now all pretty much addressed; an is¬ 
sue crept up last week related to text alerting - public hasn’t stopped getting text alerts; 
Cooper sends these alerts as email that is then converted to text for delivery by the carri- 
er/CMSP; the carriers/CMSPs are losing money on this, we think the CMSPs want peo¬ 
ple to buy text or data plan and for those with neither, CMSPs are starting to bill the 
customer; what folks don’t realize is Cooper pays a premium to the CMSPs to make sure 
they push out text alerts in bulk to the public and CMSPs don’t interpret the alerts as 
spam (RS) 

There is a small number of subscribers, target audience is very small, don’t know if they 
get the messages, don’t typically ask recipients to respond (RS) 

10. Would you please describe your process for changing how you issue alerts? What factors do 
you consider when making a change to your alert process? 

We gather feedback informally, discuss what happened after each event informally — 
feedback is huge regarding how we use the system, construct messages (KH) 

We have a very complex alerting system, desire to be consistent in how we alert; use 
formal process to implement any changes suggested via the service assessments, in¬ 
volves experimental phase, etc. (MG) 

What’s changed more than anything else is all the new channels; SM is the biggest 
change, [WEA] and IPAWS will be another (FS) 

SM reaching a different customer, so we are learning to think in terms of 140 characters 
We now have a template, we create the message in format of the primary alert medium 
selected for an incident, below that we have a space for creating the Facebook language, 
below that language for Twitter; this gives the incident commander ability to see all the 
messages going through multiple mediums will be consistent in tone and fact and he can 
approve them altogether, extract just the essential elements for the shorter messages (FS) 
In terms of level of permission to send out, that won’t change; goes from initiator to su¬ 
pervisor, chain of command, to Deputy Director of EM or Director of EM; then after 
these approvals the message is sent out; messages are always tweaked by the reviewers; 
review happens pretty quickly though, but in the middle of night things take longer (AC) 
If an alert is automatically sent by the fire department, that agency has permission to is¬ 
sue alerts for events — say, a two-alarm fire, so approval is complete within a couple 
minutes (AC) 

Next month we and sheriff’s department will be on a single system that can alert both 
land lines (reverse 911) and cell phone subscribers; we will have to make decisions 
about incorporating all the numbers (LL) 

We’ll have more accessible formats — some decision making here; goes over to another 
party to redo message in other format and then message is sent out in the accessible for¬ 
mat; no overall changes to process or how we issue alerts - all just based on timing (LL) 
Really haven’t changed; we are so far behind where we’d like to be. I’m going to be re¬ 
lying on [WEA] to fill the void for imminent life safety threats (JF) 

Still exploring mechanisms to geotarget - hoping to polygon geotargeting of our sirens; 
we are quite a ways away from getting down to alerting specifically by blocks or sub¬ 
communities (JF) 
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For weather stuff, formats, etc. we maintain those; may adjust formats slightly, our tem¬ 
plates haven’t changed in years; there are minor policy changes due to feedback from the 
public, county employees (RS) 

11. Regarding the content of an alert, what makes it effective? What makes it ineffective? Flow 
do you determine the effectiveness of your alerts? 

Relevance of the current alert, history of relevance with previous alerts (MG) 

Relevant targeting of a message, clarity of message, tell people exactly what the incident 
is and what to do about it, consistency of messages on all devices/tools used to dissemi¬ 
nate (KFl) 

Follow Dennis Mileti’s template for the most part: message label, headline, issuing 
agency (multiple partners with same message enhances trust), recommended action, con¬ 
sequences of not taking action, who should and who shouldn’t act, time to act, then 
summary and closing, where to go for more information (FS) 

No good way to determine effectiveness of alert — we just rely on hearing from one per¬ 
son or another that “this alert made a difference in my life”; our alerts just state the facts 
- blunt, to the point, issuing only when you feel it is pertinent (AC) 

Alerts should be quick, to the point, directive in nature; too much information or provid¬ 
ing opportunity for public to make own decisions is bad (LL) 

We have internal discussions to determine alerts’ efficacy; if SM is saying our alert 
doesn’t make sense, then we’d take a look at it from that perspective, too (LL) 

Try to put out alerts for life threats.. .weather events like severe, thunder, tornado, flash 
flood; we don’t put out weather warnings; we alert only for imminent immediate threats 
because we want public to understand that getting an alert means they should take ac¬ 
tion, want people to get into this mindset; when siren sounds, you need to take protective 
action, go inside (JF) 

Weather event templates were created years back in cooperation with public affairs 
(OPA) — helped with phrasing of messages - relied on them; created common message 
format used on everything across the county — “one message, many voices” (RS) 

For county employee messaging — the OPA may help us craft messages, we are always 
looking to simplify language, ease understanding; Derecho messaging was clear, benefi¬ 
cial according to public response (RS) 

12. What causes the public to ignore an alert? 

Alert is not relevant (KH) 

Alert isn’t relevant to one’s particular area — too much info to too large an area (MG) — 
not all members of public perceive a message’s relevance in the same way (incident that 
is one mile from me is relevant to some, irrelevant to others) (MG) 

Over-alerting, not having thresholds for alerting (FS) 

Won’t lose trust if an incident/threat doesn’t come to bear, if forecast didn’t verify that’s 
a good thing, tell the public what happened and why (FS) 

If you don’t close out an incident, you lose trust (FS) 

If alert is wrong, don’t think people will not react to future alerts, or they’ll just react dif¬ 
ferently (e.g., may ignore next alert) (FS) 
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Basic nature is that people will diminish the potential that they will suffer consequence 
of risk, naturally discount themselves as a statistic; we have started to alert early, pushed 
up alerting time frames, alert as early as possible so public has plenty of time to delay, 
deny, deliberate response with neighbors, and hopefully then decide to take action before 
it’s too late (FS) 

“Wasted” alerts, e.g., warning of firework noise on July 4, alerting of helicopter activity 
instead of placing a sign or message board in the actual area of helicopter activity (AC) 
Too many alerts, unclear language, no direction, too vague, unclear who sent the mes¬ 
sage; public is forgiving if you make a mistake one or two times, but they lose trust if 
you keep repeating the mistake (LL) 

Overusing the warning system for non-life-threatening situations; we want to warn only 
of imminent threats to life moving forward (JF) 

Flaving a warning system that you cannot geotarget so that you must notify an entire 
county of an incident affecting just 1 or 2 counties (JF) 

Our sirens don’t provide public with the mentality that they should go inside due to a 
chemical spill; they still think sirens are for a tornado.. .if it’s sunny then they’ll proba¬ 
bly ignore it; public has been programmed over the years that it’s a tornado siren — we 
need to change the culture (JF) 

You evacuate for a weather event, and then the weather changes course; public may 
think “the last time I left and nofhing happened, so I’ll ignore if this time” — complacen¬ 
cy is a big issue (JF) 

Too many messages is the common theme we hear; very careful to make sure there is 
added benefit when we send messages out; some subscribers have technical issues; peo¬ 
ple want more personal information, ignore messages that are too generic — but we have 
to keep the whole audience in mind; people are angered if a message doesn’t make sense 
(RS) 

13. What feedback do you get from issuing alerts, including feedback from the public? 

Calls and emails; formal service assessments; local office conducfs posf-event conversa- 
fions/interviews with community after event (MG) 

Complaints when something doesn’t work — calls didn’t go out or took too long; we see 
lack of understanding by public — very info hungry and want to know about every inci¬ 
dent; we have to play big brother and tell them only what we think they need to know 
(KH) 

SM has been the single most effective way of determining how public reacts to alerts 
and if they are even of interest to public; sometimes surprised by what does or doesn’t 
pique public’s interest; learn in real time of impact we are having (FS) 

Even if it’s an alert, lifespan is very short on SM; within first 5-10 minutes every single 
re-tweet has occurred.. .then the message is dead (FS) 

Flave had incidents in which public could send on the ground information to us faster 
than first responders on the scene - we take this as validation of our trust relationship 
with public; it has changed how we must do things; SM information must be pretty accu¬ 
rate, they aren’t afraid to correct us (FS) 
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Overall Facebook, Twitter feedback is positive; a lot of appreciation of our messaging, 
shows on Facebook through number of people who share the information, retweet — im¬ 
plies their trust; recognize we also have some disaster junkies who follow us, comment 
(FS) 

Very little feedback [here] - occasionally see increased number of people signing up, of¬ 
ten when we get media publicity; think we get so little feedback because very few peo¬ 
ple are aware of the service (32,000-35,000 subscribers) (AC) 

Wireless carriers have done fair job, but I’d guess only 2% of the public knows about 
[WEA] (AC) 

Regarding how to subscriber/opt in to service — some learn about it when they are on the 
general county website; I don’t recall seeing PSAs, bookmarks, handouts; no access to 
[WEA] promotion funding per my internet search (AC) 

Don’t think we receive any; and iCwhen we do it is negative feedback; we don’t hear 
when an alert has been useful or helpful; usually get any feedback via email (LL) 

Before we adjusted one of our systems, the public was getting winter storm warnings at 
4 AM; anyone with TV or radio would know about this anyway [media notifies people 
of weather events days early], so people started shutting devices off for this; I don’t 
think a winter storm is life threatening; a flash flood that develops quickly — that’s the 
type of warning that has to go out; most don’t want to be awakened for a watch or advi¬ 
sory (JF) 

I’m a believer, just from the bit of feedback we’ve received, in the philosophy of fine 
turning/ geotargeting alerts down to affected area—^this really will give you better public 
response; if you send an alert to people not affected by an event, you wear on people’s 
patience (JF) 

Don’t elicit much (RS) 

There is a cost associated with alerts — every message has a text fee; technical issues are 
regular due to type of phone, how they sign up; we get a lot of positive feedback (RS) 
Sometimes feedback is that the public didn’t get enough messaging, messaging wasn’t 
timely — part of the is reason we don’t have 24-hour office — I have to get to computer, 
log into the VPN, or travel to a computer - it can result in a 30-60 minute delay — we 
don’t have capacity to staff alert center 24 hours/day (RS) 

Do you collect data on the response to the alerts you issue? What data do you collect and 
how do you use it? 

Have a few formal surveys; otherwise all feedback is informal, anecdotal, minimal con¬ 
tent (KH) 

We get feedback from larger scale incidents; but they are very infrequent and each time 
they occur the system is in a different place — so don’t think you can directly correlate — 
we can detect no real pattern (KH) 

NWS collects information, passes it on to carriers (MG) 

Collect statistics on alerts sent, speed, etc.; use service assessments to gather public 
feedback (MG) 

Annual survey, questions vary by year (MG) 

Natural Hazards Center conducts studies (MG) 
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Warning Coordination Meteorologists gather regular feedback from EMs, local authori¬ 
ties; note that data is not collected for every single event nor recorded in a single place 
for all of NWS (MG) 

Haven’t collected because it’s just context of event or situation, not sure honestly what 
kind of value the data has; good for us to learn [reference is to SM feedback] what parts 
of messages people are reading (FS) 

No good way to collect data; through the vendor we can say we initiated and an alert 
was sent, but can’t confirm who in the public actually received the alert, just see vendor 
has sent it out; so there is little room for any feedback — it lies in hands of the carrier 
who actually delivered, and we don’t have a way to access the information from the car¬ 
rier (AC) 

I have limited knowledge of SM feedback (if it is collected, recorded), but nothing has 
risen to the level of saying we’re getting so much SM feedback that we have anything 
major to address; trying to work with a local university to have them be a gateway for us 
to monitor SM — with all the Crisis Common opportunities, etc. we need someone actu¬ 
ally looking at the SM information; public seems to be asking for more clarifying mes¬ 
sages rather than saying the system simply doesn’t work (LL) 

Survey monkey surveys ask very basic questions; structured more to minimize what we 
get back, we don’t have huge staff to delve into tons of data [see also response to ques¬ 
tion#!] (RS) 

Do you receive acknowledgement receipts to alerts you issue? Are the acknowledgement 
receipts analyzed by you and have they influenced how you issue alerts? 

Get an “ack” from fPAWS — not influencing current processes, just machine to machine 
communication (MG) 

No; we also do not solicit direct feedback of alerts from public, do not have staff to mon¬ 
itor public acks and are concerned they’ll communicate through this medium instead of 
911 when they are unable to respond to the alert (KH) 

Can see in real time how many have read it, haven’t read it; get an ack from alert system 
(FS) 

- No (AC) 

No, we get something from vendor on success rate of issuing messages, but no ack mes¬ 
sages from the public (LL) 

For Alert FM, we only get an “ack” that the message went out; for the emails to POCs, 
we get some responses to say they appreciate the info, but very little — we aren’t really 
asking for that, probably that’s on us (JF) 

Feedback we’ve received just strengthens that need to use the SM medium to alert (JF) 
We have the capability; but we don’t use data unless we do reverse 911 type calls - so 
we know who was home and answered the call, pass on to police department; but we 
don’t do this on a day to day basis (RS) 

We simply don’t go back and look at acks; we have asked in the RFP for more robust 
capability, function we are asking for in the next system; don’t have full time staff, and 
only have one administrator to make sure the system is working (RS) 
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16. How do you inform and educate the public regarding your alert system? How has that educa¬ 
tion affected the public’s response to your alerts? 

All just comes down to ultimately how public responds to alerts; attend public safety 
fairs, place flyers in libraries, schools, lobbies, police and fire offices; magnets; have a 
nonprofit partner who conducts outreach - all just basic outreach; no advertising budget 
(KH) 

Webinars with Warning Coordination Meteorologists (train the trainer); then then con¬ 
duct PSAs, workshops, present at schools, speak to EMs, share info on website (MG) 
NWS headlines [WEA] on its website when time/space permits; conduct seasonal cam¬ 
paigns, promote via mass media (USA Today, Today Show) (MG) 

Pushing wireless industry to conduct [WEA] outreach (MG) 

There is synergy between alerts, regional joint information center website, SM, press 
conferences; Facebook and Twitter icons part of alerts; website popups; promotion lan¬ 
guage part of alert template; capitalize on these synergies during hurricane season (FS) 
We really don’t inform or educate; we’ve gone to lengths to get funding from the county 
to educate the public about our services, but the budget is very tight; every once in a 
while, if folks are out doing public service or a community event, we’ll promote; ask 
public to opt in during a press event (AC) 

Almost takes an incident for public to get onboard, to sign up; I’ve been following this 
trend for years in every system I’ve worked - Super Bowl, convention, etc. (AC) 

We do a number of education activities - promote during actual events (most helpful), 
reminders on press releases, partner with businesses and nonprofits to do incentivized 
promotions (car dealerships. Girl Scouts, Papa Johns); media buys; booths at community 
events, presentations at rotary meetings and local businesses; message is the same: get a 
plan, know what you need to do, register your phone, get a kit, stay informed (LL) 

In terms of effect to public response: does well, have 200,000 people opted into alert 
service; DHS studies and Centers of Excellence have recognized the efficacy of our opt 
in raters — think we have one of highest rates in the United States (LL) 

We have a media campaign to encourage people to purchase an Alert FM device; in 
event of an emergency situation, PSAs, reminders, getting on radio, reassuring and 
providing additional info via media (JF) 

I’d turn the question on you - how does any education really work? Some appreciate it, 
some expect you to be at their door in 5 minutes.. .would like to think getting feedback 
from the public is a positive thing. You get a lot of “I saw you on TV or radio the other 
day” feedback. We know people are hearing it. (JF) 

In 2007, one question related to public education was a positive rating response regard¬ 
ing our job of educating the public about risks and hazards.. .we are trying to make peo¬ 
ple self-sustaining rather than just relying on us (JF) 

Our attrition rate is low on the subscriber base, which increases approximately 
10,000/year - we conduct outreach like crazy to attract subscribers; promote hard radio 
system; NOAA radio; distributed 650 radios throughout county at an event; share com¬ 
mon preparation message, common alerting and communications — have radios, batter- 
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ies, flashlights; earthquake drill — did messaging for this; tornado drill — provide a full 
presentation, messaging to all county residents (RS) 

Conduct dedicated individual outreach - any community tunction, conduct business out¬ 
reach, tabletops, always use the canned message “make a plan” and include messaging 
about signing up for CEAN; want them to know the NCR blog is out there, can get info 
about Fairfax and the NCR; we also use a county blog (RS) 

Regarding [WEA] — I have an article that is almost done — we have working group, 
YouTube, plan to blog to message and let people know of major carriers participating in 
[WEA], what message will look like, trying to get the region on board, public affairs to 
push out messages and let people in region know [WEA] is out there, but must be care¬ 
ful with messaging - only [a few jurisdictions] have it.(RS) 

17. What should be done to improve alert systems? What would be the perfect alert system? 

We need proximity based alerting (MG) 

Want to see following changes (ranked) (MG): 

a. Improve geotargeting 

b. Proximity based alerts 

c. Graphic of warned area 

d. Embedded link to additional info (graphics, additional detail) 

e. Allowing opt-in/out of Severe and Extreme alerts (note: implementation of carriers 
is not exactly consistent — e.g., iPhone has all or none for opting in and out) 

f. Increase text length to traditional text messaging limit from 90 characters 

g. Consistent name for service 

h. Remove Confusing Text below WEA Message 

i. WEA service indicator (suggested by FEMA, like wireless emergency alert service 
indicator so you know if area can actually send a [WEA] alert to you) 

Fluge public expectation management issue (KFI) 

[WEA]: geotargeting and 90-character limitations; also just learning about option to is¬ 
sue free form [WEA] alert messages (KFI) 

Alerts can start to become more effective by personalizing them — alerting on personal 
devices solves targeting, can overcome translation issue and disabilities (e.g., blind per¬ 
son’s or Spanish speaker’s devices will display in way that user needs), users can set 
level of alerts they want to see (though I am not a fan of that — think this should be up to 
EMs, not public) (KFI) 

Only perfect system would be a chip in the head, driven by satellite, but then you still 
have single point of failure. There is no one perfect system — you need a layered system 
and multiple methods because people aren’t always doing the same thing, they moving 
around in the car, home, at work, away from home on vacation in another part of country 
— all sorts of things you need to communicate information (MG) 

Need to ensure consistent messaging (MG) 

A lot of challenges to work on; in a perfect system, you can communicate with all peo¬ 
ple, channels, times, all hazards (FS) 
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Need better integration of all systems, mobile technology (Pandora, apps, additional 
bandwidth to cover supplemental information) (FS) 

90-character limit of [WEA] is intimidating; are we doing ourselves a favor or creating a 
problem by limiting amount of data we can provide? (FS) 

System that compiles data for you, allows public to respond to alerts and ask questions, 
and is monitored by staff; someone can issue an alert on the fly using an app, able to ex¬ 
port data to excel, data can be made as interchangeable as possible (AC) 

Automatic alerts — right now, when a tornado warning is issued, an auto alert is sent out 
to the public; if automatic, there is less chance of error (AC) 

Whole idea of [WEA] would be perfect with lack of opt-in requirement; ability to alert 
in multiple languages, get out messages faster, include graphics, etc. in messages (LL) 
Regarding 90-character limit - very low — why not increase when carriers say they can 
go up to 160? (LL) 

Need nationwide public education and need funding to do it, especially if you have to 
opt in 

County level geotargeting of [WEA] great if your county is small.. .but here in Califor¬ 
nia, the counties are much larger whereas east of Mississippi areas are more city-centric 
(LL) 

There should be one point of distribution to several alert systems with a strong focus on 
geotargeting (JF) 

Parameters of use are important; you can twist any event to say it is somehow life threat¬ 
ening; several years ago we had a tabletop event in Chicago and talked about FI INI; the 
health department would send out alerts every hour (JF) 

1 have been onboard with [WEA] for years — it’s one of the ways to go, will provide 
great opportunity — would like to see it hit cell phones and email - email would be an 
opt in piece (RS) 

No cost to citizens (RS) 

Full functionality allowing me to drill down to cell tower and it tells me the exact cover¬ 
age/true alert area (RS) 

Messages longer than 90 characters — slight increase would be beneficial for [WEA] 

(RS) 

CMSPs should adverfise a lof when you buy a phone; people in this region are already 
getting [WEA] messages and have no idea what they are — they call us all the time to ask 
(RS) 

NWS alerts should be automatic — if an event/reading hits certain criteria, an alert should 
be automatically routed to [WEA] (RS) 

Full mobile capability - if I’m driving down road, 1 want to pull over and have my i- 
Phone give me access to send a message with full functionality — this would be huge 
(RS) 

Geotargeting capability— Google Earth to really drill down (RS) 

All the real numbers within a geographic database — worldwide system (RS) 
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Version has an e911 database; but there is so much transition, the list is made outdated 
daily; unless you get daily update your list is out of date - want this to auto populate into 
my database — hit cell phones and home phone and provide language translation (RS) 
Closed captioning on phones to alert the hearing impaired — have a special vibration, 
visuals (RS) 

I’d like to be notified if someone else issues alerts; I want to know (RS) 

Ability to manage through WebEOC software used by NCR (RS) 

Report tracking — find out how many people are actually getting messages, proactive ca¬ 
pability, see who has gotten it (RS) 

18. Have you consulted with anyone regarding ways to make your alerting system more effec¬ 
tive? Who? 

Looking at Sprint blog — consistent with what we’ve seen in scientific lit - public wants 
relevant messages, will disable device if annoyed by messages; communicate with other 
agencies, vendor, public safety forums, etc. for lessons learned (KH) 

Working one on one with carriers, AXIS standards organization, FCC (MG) 

Our vendor. Emergency Management magazine blog, Linkedin mass notification forum 
(AC) 

We’ve done some public forums to find out what recipients would like to be notified of. 
Have spoken with POCs that we can alert versus also notifying the public; we’ve con¬ 
ducted cross border discussions to make that better (JF) 

Still think there’s a huge gap in alerting across America if you are going to do it right. 
Sure you can send out alerts, but does the public really understand and respond to the 
current systems? (JF) 

In response to the RFI this summer — 6 companies showed their products to us — a lot of 
neat ideas that we incorporated into our proposal request; also reviewed some studies; 
we plan to conduct site visits to see systems in the region, what they do, and how well 
users like them; don’t want to do this acquisition process again for a long time (RS) 

19. How does [WEA] figure into your alert strategy? What do you see as benefits and detrac¬ 
tions of using [WEA]? 

Biggest benefit of [WEA] is that it doesn’t require public outreach, subscribers; public 
knows about system to receive the alert — huge advantage (KH) 

Can’t be used for local incidents — no way to tailor the message for all the people in the 
huge geographic area being targeted (KH) 

People respond well to texts — great force multiplier to messages they get via telephone 
(KH) 

[WEA] doesn’t replace other systems, is part of a larger network (MG) 

[WEA] will be added to our template, need to integrate into systems/services (FS) 
Benefit of [WEA] is reaching all our constituents, all our commuters; but in some ways 
[WEA] is worse than EAS — people aren’t accustomed to getting the [WEA] message — 
until it’s an accepted part of how emergency management communications with the pub¬ 
lic, we must use it judiciously, first few messages will generate heightened sense of in¬ 
terest (FS) 

Assume we’ll be able to alert at cell-tower level eventually (FS) 
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Regarding regional alerting, not just within a region now, need to create con¬ 
sistent/uniform alert triggers, formatting style, concept of how we issue alerts, close out 
alerts — all this must be part of an overall communications strategy to build credit, trust, 
etc. (FS) 

Thresholds probably higher than EAS for [WEA] until we find a good fit for [WEA], 
EAS and [WEA] will probably be used for same incidents in future; this is initial ap¬ 
proach until public acclimates and we pass learning curve; plan to take a lot of lessons 
learned from NWS’s use of [WEA] (FS) 

[WEA] fits completely into my alerting authority, but we are cautious about when to use 
it unless it is for a large specific emergency event; obstacle is it can’t be issued for small 
areas; thought of using it for tropical storms, but I want public to know about the service 
first; if we issue county-wide emergency messages, a [WEA] alert would fit right in 
(AC) 

[WEA] ability to communicate via cell towers helpful in reaching [tourists], frees us 
from dependency on subscriptions to send geotargeted alerts... [WEA] is another tool to 
fit into our service to geotarget - however, [WEA] is county-wide, don’t want to alert 
those two hours away (LL) 

[WEA] is important if used for the right purpose and especially if its messages can be 
geotargeted; I think [WEA] is invaluable, especially when you get to the point of target¬ 
ing cell towers - especially when you can contact mobile device users and those passing 
through your county (JF) 

[WEA] alerts from NWS will be automatic — we’ll try to coordinate so we know it went 
out — but we’re still going to duplicate efforts - want to ensure message is the same, ha¬ 
ven’t sat down to work this process out with NWS yet (RS) 

When sending only to cell phones, the message goes out, is rebroadcast every x minutes 

— you have so much transition, people will drive into an area, get the alert, then leave the 
area and not know finality, not get an event completion message; not sure they will get 
messages when traveling through the region in between broadcasts (RS) 

For the NCR, we have people who speak different languages — this is a concern with 
[WEA] (RS) 

If I send an alert to the area that borders [neighboring counties] — how do I deal with this 
bleedover? How do I coordinate? Every jurisdiction wants to send its own messages — 
don’t expect a single entity will be accepted as messaging on behalf of the whole county 

- never (RS) 

Don’t know which CMSPs can geotarget down to cell tower level (RS) 

Regarding the character limit, planning to create templates in advance to fill in the blank 
(RS) 

Can only issue [WEA] alerts from one location, the EM net desktop PC in our office — 
so I must be in that room to send it; Web EOC is working on this issue along with 
Cooper to provide more flexibility so we can just log into the system from home (RS) 
[WEA] is very beneficial but we have to be even more careful of crying wolf and people 
opting out - then we’ll lose our big audience (RS) 
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On Monday, 11/5 call with SRA [on behalf of SEl’s [WEA] Systems Integration Project] 
— mentioned 1 want to connect with the CMSPs and find out what geotargeting capabili¬ 
ties they have, which phones are [WEA] capable — but they are squeamish, don’t want to 
tell us about their capabilities (RS) 

20. Please describe any experience you’ve had with [WEA]? How has that worked? Was it ef¬ 
fective? 

[WEA] alerts are delivered quickly, feedback is good in larger cities, poorer in rural are¬ 
as (MG) 

Grappling with need to test system — more important to AO than public — but you can 
overtest (MG) 

We don’t use [WEA] — targets too large an area and can’t tailor messages; even EAS 
provides us more flexibility of messages compared to [WEA] (KH) 

Huge disconnect by FEMA in not telling people about option to issue free form alerts 
(KH) 

Have yet to use [WEA], determining best way to use it and erring on the side of caution; 
system is ready here, once 1 use it for a live event, 1 won’t need to participate in S&T’s 
testing [note: that Miami-Dade is on the list of alert originators who have agreed to par¬ 
ticipate in the first regional test being conducted by DHS/SEl/SRA] (AC) 

[We have] permission from FEMA to issue free-form [WEA] alert messages, short code; 
do not have to rely on CAP-extracted elements to issue a [WEA] alert (AC) 

We haven’t used it; all experience is from NWS sending messages and our playing catch 
up, responding to media, people asking us why they are getting evacuation notices when 
it is sunny in their part of the county; experience has been negative so far because our 
experience has been reactionary, we are worried about the public already opting out due 
to the issues with NWS messages in this area (LL) 

We’ve done all we needed to adopt so far, filled out paperwork, tested with Sprint, etc. 
but our vendor is not able to send [WEA] alerts yet; that said, not sure we’d use [WEA] 
based on only being able to alert using county FIPS codes — there’s nothing we can send 
out that would be a county-wide message (LL) 

Haven’t used [WEA] — that’s the frustrating part [waiting for first Regional Test] (JF) 
Like that you have to opt out not in, think that’s huge; if abused, you’ll get a lot of opt- 
outs though (JF) 

Regarding plans for [WEA] use — we’ll try to be consistent with Alert FM as far as 
weather is concerned; not sure if NWS will have access or perhaps we’ll be able to cap¬ 
ture their messages; use for evacuations, in-place sheltering (JF) 

Hoping there will be some policy for weather use; would hate to see system abused on 
important but not life-threatening situations (JF) 

[DHS staff] visited the [regional authorities] several months back; 1 went to Las Vegas in 
Febraary to present on [our] ideas; worked with my state representative to get all this 
[WEA] adoption work done; already received a COG, etc.; the state is having issues be¬ 
cause they have a committee that wants us to send EAS alerts along with [WEA] alerts — 
state isn’t going to tell us what to do (RS) 

I’ve taken the FEMA IPAWS course online (RS) 
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Waiting for Web EOC and Cooper to come up with their hook, planning conversations 
with fire and police on best ways to use [WEA] (RS) 

I know of discussions of version 3 and next generation — I’d love to see the federal part¬ 
ners put together a big campaign for [WEA] (RS) 

Additional Questions: 

• Are there any factors we’ve missed that you believe influence the public’s trust in alerts? 

Lack of trust in system doesn’t preclude me from using system, important to work close¬ 
ly with vendor to get your system right (KH) 

Even if system works perfectly, you still need public trust - if people don’t understand 
your procedures, don’t trust your agency, your system will be ineffective - and unfortu¬ 
nately you can’t reach your entire constituency to communicate all this information to 
them (KH) 

Outreach is incredibly important — public should know whom to contact with a problem 
(MG) 

Carriers still have a lot of work to do, still sending callers back to us with questions 
(MG) 

Building expectations with the public and living up to them is key — requires a ton of 
communication with public throughout process (MG) 

We need to factor into bust model how to communicate a mea culpa to public — know 
there will be some mistake, some erroneous message that goes out — how should AOs 
handle that? (MG) 

Alerts should be issued to the most precise area possible and understand exactly who 
will get that alert (MG) 

Must ensure 911 is at the table when making alerting decisions, keep 911 in the loop 
when alerts go out; recognize difficulty of this because 911 is slotted within different 
parts of public alerting (EMAs, local law enforcement.. .so easy to think someone else is 
in communication with them, and ultimately they are left out) if they aren’t in the loop 
and members of the public call 911 about an alert, they will have less bust in the alert: 

“if 911 doesn’t know about it, how can we believe it?” (LF) 

You are stumbling on concept of tmst — singular variable — hunch that what you’re after 
is a little more complex; AO trust in [WEA] is separate and distinct from public response 
to alerts (DM) 

I don’t think [AOs] assess [public tmst], try to predict it, or even think about that (DM) 
Scientific literature about public response has an incredibly detailed synthesis about fac¬ 
tors that bear weight on tmst — independent variables — tmst is just one of a dozen you 
must take into account to figure out public response; you can’t predict response from just 
the variable of trust (DM) 

I’ll bet there are people in the country freaked out by CAP, IPAWS and [WEA] — and all 
the proposals, money, etc. they need to expend to get up and running with [WEA] - 
think it might be useful to measure some control on it — originator’s response to adopting 
[WEA] (cost, intimidation of process), to the system itself must be assessed and factored 
in somehow (DM) 
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without doing a national education campaign to at least the big cities about the program, 
its guiding principles, etc., [WEA] won’t be very effective.. .at least until a disaster oc¬ 
curs and people learn more about it; most counties depending on state to educate, state 
on Feds to do it.. .or carriers or other system owners; forget the emails — spend 44 cents 
and send a hard letter to every county EMA director or manager (AC) 

The word isn’t getting out — people don’t have time or expertise to do the paperwork; if 
FEMA would provide someone from each region as an assef fo the states in all 10 re¬ 
gions, you’d have way more people signed up for [WEA] (AC) 

Are there any factors that influence your use of an alert system, either positive or negative, 
that we have not covered? 

Who owns education? Who is responsible? Train has left the left station, public has no 
idea what we are doing re: [WEA] (LL) 

We are a border county — messages have potential [bleedover] to cross not only a county 
or state line but international boundaries (LL) 

Would you be willing to discuss this topic further with us? We would like to get more in- 
depth information about your experiences. 

DM: Yes, prefer calls over document reviews 

- MG: Yes 

KH: Yes through end of 2012; KH to provide new POC 

LF: Yes, but probably not best resources; LF to provide new POCs 

- FS: Yes 

- AC: Yes 

- LL: Yes 

- JF: Yes 

- RS: Yes 
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Appendix C Original List of Factors Considered for Public 
Trust Model 


Proposed Public Trust Measures for the WEA Trust Model 

1. Relevance of alerts 

2. Public interest in subscribing to alerts 

3. Promotion of the alert system to the public 

4. Opt-out rates 

5. Criteria for issuing alerts 

6. Criteria for amending alerts 

7. Frequency of alerts 

8. Issuing a final communication 

9. Disconnect between public versus official’s use of alert system 

10. Alerts that communicate action to take 

11. Alert originator direct engagement with public in advance of issuing alerts 

12. Use of a public notification office 

13. Translation of alert into multiple languages 

14. Public control over the rendering of alerts on their devices 

15. Easy public “follow us” mechanisms 

16. Types of threats in alert system 

17. Public selection of type of alert to follow 

18. Public selection of time window to receive alerts 

19. Brevity of message so public can read quickly 

20. History of relevance 

21. Clarity of message 

22. Consequence of no action 

23. Who should vs. should not act 

24. Time window to act 

25. Summary and closing 

26. Venue to seek more information 

27. Provision of instructions (the public need not make guesses) 

28. Public perception of relevance 

29. Over alerting 

30. Messages that tell public what happened and why 

31. Psychology to underestimate threats and overestimate benefits 

32. Lead time provided by alert 

33. Degree of wasted alerts 
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34. Repeated violations of trust that cause cumulative damage 

35. Public perception of info withheld 

36. Public use of communication mechanisms to share real-time info 

37. Degree to which public overtly corrects alert originators 

38. Degree to which public shares or retweets information 

39. Ability for public to request clarifying information 

40. Degree to which the alert is personalized 

41. Customization by location, time, type of alert 

42. Degree to which the public can provide feedback and feel heard 

43. Public perception of government 

44. Degree of redundancy of alerting 

45. Public location to charge device versus location for which public wants to be notified 

46. Degree to which transients and visitors are handled in a given area 

47. Degree to which public can interact with alert system 

48. Degree to which alerts are viewed as spam 

49. Degree to which alerts do not address imminent life-threatening situations 

50. Degree to which alert does not make sense 

51. Degree to which 24-hour coverage is provided by alert system 

52. Degree of outreach, giveaways, TV presence 

53. Degree system is promoted at time of sale of cell phone 

54. Closed captioning and special vibration 

55. Degree to which different local jurisdictions handle alerts in uncoordinated fashion 

56. Imminence of threat 
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Appendix D Public Trust Model Factor Descriptions 


Factor 

Description 

1_Relevance 

Applicability of the alert to the receiver. Does it affect the receiver’s 
current location? Is it received at the appropriate time? ... 

10 Action to take 

A definitive statement of action to be taken 

100 Believing 

Recipient accepts the alert as true 

101 Hearing 

Recipient receives and reads the alert 

103 Acting 

Recipient takes action stated in the alert 

103 Understanding 

Recipient comprehends the information provided in the alert 

12_Alert source 

The governmental tier of the sender (i.e., local, county, state, fed¬ 
eral) 

15_Easy additional follow-us mechanisms 

Ease of obtaining additional information from the sender via other 
communications channels 

20 History of relevance 

The applicability of previously received alerts to the recipient 

21 Clarity of message spelling and grammar 

The degree of grammar and spelling errors in the alert 

23_Who should act 

A definitive statement of which recipients should take the actions 
stated in the alert 

24_Time window to act 

A definitive statement of when the recipient should take the actions 
stated in the alert 

26_Where to go for more information 

A definitive statement of places to seek additional information re¬ 
garding the event precipitating the alert 

3_Public awareness of WEA 

Public knowledge of WEA prior to issuance of an alert, developed 
through outreach via media channels (TV news reports, radio news 
reports, newspaper stories) 

30 Explain what has happened 

A definitive statement of the event that has precipitated the alert 

32_Lead time provided 

The amount of time between the issuance of the alert and the mo¬ 
ment when action must be taken 

33 Degree of wasted alerts 


37_Confirmation via social media 

Information contained in the alert is disseminated by others through 
social media networks such as Facebook and Twitter 

4_Opt-out rate 

The percentage of alert receivers who choose to disable the receipt 
of future alerts 

44_Redundancy of alerting 

Information contained in the alert is also available through other 
channels such as TV and radio news 

48 Alerts viewed as spam 

Alerts are pre-judged as spam 

55_Local jurisdictions act uncoordinated 

The level of cooperation between senders within a region, as evi¬ 
denced by avoidance of redundant alerting, agreement between 
alerts, etc. 

7 Frequency 

The time rate at which alerts are received (e.g., alerts/month) 

70 Explain why I should act 

Provides a justification for the action stated in the alert 

71 Message in primary language 

Alert is provided in the primary language of the receiver 

8_History of final communication 

Issuance of a final communication (e.g., all-clear notice) at the end 
of the event 

99_Type of alert 

Presidential, Imminent Threat, or AMBER 


CMU/SEI-2013-SR-021 | 82 
































Appendix E Alert Originator Trust Model Factor Descriptions 


Factor 

Definition 

Appropriateness 

The degree to which WEA provides an alerting solution that is appropri¬ 
ate to the event 

Authority 

Permission and prerogative of the AO to issue the alert 

Certainty 

The verifiability of the associated event is sufficient to justify a WEA 
message 

Geographic breadth 

The size and location of the geographic region impacted by the emer¬ 
gency event is consistent with WEA capabilities 

Responsibility 

The AO’s obligation and authority to issue the alert (i.e., is it clear that 
the responsibility and authority to issue the alert resides with the AO, or 
could some other organizations be responsible for issuing the alert?) 

Severity 

The degree of impact associated with an event is consistent with WEA 
usage 

Time of day 

The time of day (e.g., waking hours, middle of the night) when the alert 
is to be issued 

Urgency 

The degree of immediacy associated with an event is consistent with 

WEA usage 

Availability 

The degree to which the WEA system is capable of being used when 
needed to issue an alert 

Security 

The degree of confidence that the WEA service is robust against at¬ 
tempted cyber attacks (e.g., spoofing, tampering, and denial-of-service 
attacks). 

System accessibility 

The ability of AOs to gain access and admittance to the WEA service 
when and where desired 

Remote/portable access 

The ability of AOs to generate WEA messages from remote locations 

System ease of use 

The facility (or difficulty) with which AOs may use the WEA service to 
issue alerts 

Cross-system integration 

The ability of the WEA service to work in conjunction with other emer¬ 
gency management systems 

Magnitude of effort 

The amount of time and work needed to issue the alert 

Templates 

The availability of predefined formats and information to accelerate and 
ease the process of alert issuance 

System readiness 

The degree to which the WEA service is operable and ready for use 
when needed 

System reliability 

The degree to which AOs may depend on the WEA system to operate 
correctly when needed 

Training 

Creation of skills, competencies, and knowledge for AOs 

Practice 

The exercising of skills needed to operate the WEA service effectively 

Skills/competencies 

The aptitude and capability to operate the WEA service effectively 

Understanding 

The knowledge of the operational characteristics of the WEA service 

Effectiveness 

The degree to which the WEA service accomplishes its intended pur¬ 
pose 

Accuracy 

The ability of the WEA system to disseminate correct alert information 
to intended recipients 

Location accuracy 

The ability of the WEA service to disseminate alerts to the defined loca¬ 
tions 

Message accuracy 

The ability of the WEA service to disseminate alerts with the message 
content intended by the AO 

After-action review data 

Knowledge resulting from in-house review and analysis of prior WEA 
message disseminations 
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Factor 

Definition 

Alert frequency 

The number of WEA messages issued within an area in the immediate 
past 

Message understandability 

The ability to convey necessary information within the constraints of the 
WEA message 

Public awareness/outreach 

The establishment of prior awareness and public education regarding 
WEA services 

Public feedback history 

Information received from the public regarding prior WEA messages 
(e.g., “thanks for warning me,” “don’t wake me at night”) 

System feedback 

The quality and value of information describing system function that is 
provided by the WEA service to the AO 

Historical system feedback 

Information from the WEA service regarding prior performance (e.g., 
dissemination time, alert geolocation data) 

Real-time system feedback 

Information from the WEA service reporting the status of the current 

WEA message dissemination process (e.g., message delivered, mes¬ 
sage rejected) 

Timeliness 

The ability of the WEA service to disseminate a WEA message within a 
suitable time frame 
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Appendix F Relationships Among Public Trust Model 
Factors 


Factor / Hierarchy 

103_Acting 

10_Action to take 
24_Time window to act 
32_Lead time provided 
23_Who should act 
1_Relevance 

30_Explain what has happened 
70_Why I should act 
23_Who should act 
100_Believing 

30_Explain what has happened 
70_Explain why I should act 
44_Redundancy of alerting 
33_Degree of wasted alerts 
20_History of relevance 
1_Relevance 

30_Explain what has happened 
70_Explain why I should act 
23_Who should act 

55_Local jurisdictions act uncoordinated 
3_Public awareness of WEA 
8_History of final communication 
12_Alert source 
7_Frequency 

21_Clarity of message spelling and grammar 
37_Confirmation via social media 

55_Local jurisdictions act uncoordinated 
26_Where to go for more information 
103_Understanding 

55_Local jurisdictions act uncoordinated 
21_Clarity of message spelling and grammar 
71_Message in primary language 
15_Easy additional follow-us mechanisms 
101_Hearing 

4_Opt-out rate 

55_Local jurisdictions act uncoordinated 
44_Redundancy of alerting 
48_Alerts viewed as spam 
99_Type of alert 
7_Frequency 

3_Public awareness of WEA 
20_History of relevance 
1_Relevance 

30_Explain what has happened 
70_Explain why I should act 
23_Who should act 

21_Clarity of message spelling and grammar 
44_Redundancy of alerting 
33_Degree of wasted alerts 
48_ Alerts viewed as spam 
99_Type of alert 
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7_Frequency 

3_Public awareness of WEA 
20_History of relevance 
1_Relevance 

30_Explain what has happened 
70_Explain why I should act 
23_Who should act 

21_Clarity of message spelling and grammar 
44_Redundancy of alerting 
33_Degree of wasted alerts 
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Appendix G Relationships Among Alert Originator Trust 
Model Factors 


Factor / Hierarchy 

WEA utilization 

Appropriateness 

Urgency 

Severity 

Certainty 

Geographic breadth 
Time of day 
Consequences 
Responsibility 
Authority 
Availability 

System readiness 
System accessibility 

Remote/portable access 
System reliability 
System ease of use 

Magnitude of effort 
Cross-system integration 
Templates 

Training 

Skills/competencies 

Understanding 

Practice 

Security 

Effectiveness 

System feedback 

Real-time system feedback 
Historical system feedback 
Public feedback history 
After-action review data 
Timeliness of dissemination 
Message understandability 
Accuracy 

Message accuracy 
Location accuracy 
Public awareness/outreach 
Alert frequency 
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Appendix H Public Trust Model 





Figure 9: Public Trust Model Condensed 


CMU/SEI-2013-SR-021 | 88 










Combo 13 


- * Btllavingpar * 

^1_CI»»y_llMt«g>, \ Aw«renci. • 

_Spalllng_Grammar ) % ^ 

^ .Content J _'V 

S.Public ^■ren.t>\ ♦'eeiieving per'' Believing per 

_of_WEA J I Reieyjnce • Wasted Alerts 

% t 

* Spam per ' 

• Awareness * 

' / » • » 

^ m m '' /4o_History_Of_ 

^ * Spam per ^ Relevance 

• Relevance * \ 

Combo 04 


' !) 

A * Believing per * ^ " ""C 

\ * More Info * * ' 

I Act per Action ^ 

__ ■" * ' 


Combo 12 


' Spam per Alert ' 
\ Typ. ^ 





* 1 .. 

* ActperLead ^ . . t. '4 

* Time •; Actpernme * 

\ 0 ' Window * 



Figure 10: Public Trust Model Expanded 


CMU/SEI-2013-SR-021 | 89 










Appendix I Alert Originator Trust Model 
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Figure 11: Alert Originator Trust Model Condensed 
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Appendix J Public Surveys 


Public Survey 1 

Q1: A Survey for the Wireless Emergency Alert Service 
WEA Background 

The Wireless Emergency Alerts (WEA) service is a partnership between the FCC, FEMA, and 
wireless carriers, whose purpose is to enhance public safety. WEA enables authorized federal, 
state, and local government authorities to send geographically targeted, text-like alerts to the pub¬ 
lic via wireless mobile devices such as cell phones and tablets. This service can deliver alerts from 
the president of the United States, AMBER alerts, and alerts regarding imminent local threats 
such as floods, chemical spills, civil unrest, etc., to all compatible mobile devices within the geo¬ 
graphic area specified by the originator of the alert. 

On compatible mobile devices, WEA uses a unique signal and vibration to attract attention. WEA 
messages will not have to be opened like SMS text messages, but will pop up on the device’s 
screen. The public does not need to sign up to receive WEA messages — participating cell carriers 
will sell WEA-capable phones with the service already opted-in, and there is no charge for the 
WEA service or the messages you receive. 

This Survey 

The purpose of this survey is to help government authorities use WEA in a manner that enhances 
public safety. For WEA to be effective, alert originators must be able to issue alerts that are re¬ 
ceived, understood, believed, and acted upon by the public. This survey explores several factors 
that influence public response to alerts. 

Q2: Online Consent 

This survey is part of a research study conducted by Robert W. Stoddard II at Carnegie Mellon 
University. The purpose of the research is to study the factors that may influence the public’s or 
alert originator’s trust in the soon-to-be-deployed United States Wireless Emergency Alert (WEA) 
notification system. 

Procedures 

Survey recipients will be asked approximately 20 multiple choice questions regarding their reac¬ 
tions to different aspects and content of potential alert messages. The survey is expected to take 
no more than 5-7 minutes. 

Participant Requirements 

Participation in this study is limited to individuals age 18 and older. Participants must currently 
use some form of a cellular phone. 
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Risks 


The discomfort and risk associated with participation in this study are no greater than those ordi¬ 
narily encountered in daily life or during other online activities. Discomfort may arise from the 
need to maintain focus and attention when responding to the 20 multiple choice questions that 
include repetition and subtle differences within the message content. The risk of a breach of con¬ 
fidentiality resulting in the inadvertent release of email addresses will be managed by the re¬ 
searcher by ensuring secure access to this information through password control of the online 
survey tool, subsequent removal of the information from the online system, and archival storage 
in a secured filing cabinet under control of the researcher. 

Benefits 

There may be no personal benefit from your participation in the study but the knowledge received 
may be of value to humanity. Respondents will be notified of a future, free, invitation-only webi¬ 
nar providing more detail of the Wireless Emergency Alert system. 

Compensation and Costs 

There is no compensation for participation in this study. There will be no cost to you if you partic¬ 
ipate in this study. 

Confidentiality 

By participating in this research, you understand and agree that Carnegie Mellon may be required 
to disclose your consent form, data, and other personally identifiable information as required by 
law, regulation, subpoena, or court order. Otherwise, your confidentiality will be maintained in 
the following manner: Your data and consent form will be kept separate. Your consent form will 
be stored in a locked location on Carnegie Mellon property and will not be disclosed to third par¬ 
ties. By participating, you understand and agree that the data and information gathered during this 
study may be used by Carnegie Mellon and published and/or disclosed by Carnegie Mellon to 
others outside of Carnegie Mellon. However, your name, address, contact information and other 
direct personal identifiers in your consent form will not be mentioned in any such publication or 
dissemination of the research data and/or results by Carnegie Mellon. 

Right to Ask Questions & Contact Information 

If you have any questions about this study, you should feel free to ask them by contacting the 
Principal Investigator now at Robert W. Stoddard II, Principal Researcher, Software Engineering 
Institute, Carnegie Mellon University, 4500 Fifth Avenue (Office 3110), Pittsburgh, PA 15213, 
412-268-1121, rws@sei.cmu.edu. If you have questions later, desire additional information, or 
wish to withdraw your participation please contact the Principle Investigator by mail, phone, or e- 
mail in accordance with the contact information listed above. 

If you have questions pertaining to your rights as a research participant; or to report objections to 
this study, you should contact the Research Regulatory Compliance Office at Carnegie Mellon 
University. Email: irb-review@andrew.cmu.edu. Phone: 412-268-1901 or 412-268-5460. 
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Voluntary Participation 

Your participation in this research is voluntary. You may discontinue participation at any time 
during the research activity. 

Q3: Please answer the following three questions related to your consent. If you answer “No” to 
any question, you will be ineligible to take the survey. _ 



Yes (1) 

No (2) 

Are you age 18 or older? (1) 

o 

o 

Have you read and do you understand the previous online consent information? (2) 

o 

o 

Do you want to participate in this research and continue with the survey? (3) 

o 

o 


NOTE: Q4 through Q24 are presented only if the respondent answers “Yes” 

to all questions in Q3. 


Q4: Instructions 

Each of the following survey questions presents you with a situation, and asks you to estimate the 
likelihood of a particular response to that situation. The choices for your answer are 
Definitely Not I would respond to the situation less than 5% of the time 
Very Probably Not I would respond to the situation 5-20% of the time 
Probably Not I would respond to the situation 20^0% of the time 

Undecided I would respond to the situation 40-60% of the time 

Probably I would respond to the situation 60-80% of the time 

Very Probably I would respond to the situation 80-95% of the time 

Definitely 1 would respond to the situation more than 95% of the time 

Q5: How likely are you to believe a WEA message if you determine that the message is not rele¬ 
vant to you (e.g., it does not apply to your location, it is not issued in a timely manner, or it does 
not address an emergency that will affect you)? 

o Definitely Not (1) 

o Very Probably Not (2) 

o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 
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Q6: At 8:05 PM, while in the town of Kent, your cell phone receives and displays the following 
WEA message: “Hazardous Materials Warning in this area until 08:49 PM. Take shelter now. 

Kent Fire Dept.” How likely are you to take action in response to this message? 

o Definitely Not (1) 

o Very Probably Not (2) 

o Probably Not (3) 
o Undecided (4) 

o Probably (5) 

o Very Probably (6) 

o Definitely (7) 

Q7: At 8:05 PM, your cell phone receives and displays the following WEA message: “Fire 
Warning in this area until 08:49 PM. Evacuate now State Police.” How likely are you to believe 
this message? 

o Definitely Not (1) 

o Very Probably Not (2) 

o Probably Not (3) 
o Undecided (4) 

o Probably (5) 

o Very Probably (6) 

o Definitely (7) 

Q8: How likely are you to believe a WEA message if you have received prior WEA messages that 
have not been relevant to you (e.g., they did not apply to your location, they were not issued in a 
timely manner, or they did not address an emergency that affected you)? 

o Definitely Not (1) 

o Very Probably Not (2) 

o Probably Not (3) 
o Undecided (4) 

o Probably (5) 

o Very Probably (6) 

o Definitely (7) 
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Q9: At 8:05 PM, your cell phone receives and displays the following WEA message: “Hazardous 
Materials Warning in this area until 08:49 PM. Take shelter now.” How likely are you to under¬ 
stand this message? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q10: At 8:05 PM, your cell phone receives and displays the following WEA message: “Hazerdus 
Mateirals Warning in this area until 08:49 PM. Take shelter now.” How likely are you to view this 
message as spam? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q11: You are in ZIP code 12345 in the town of Kent when your cell phone receives and displays 
the following WEA message: “Hazardous Materials Warning until 08:49 PM. Take shelter now. 
Kent Fire Dept.” How likely are you to view this message as relevant to you? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 
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Q12: You are in ZIP code 12345 in the town of Kent when your cell phone receives and displays 
the following WEA message: “Hazardous Materials Warning until 08:49 PM. Take shelter now. 
Kent Fire Dept.” How likely are you to take action in response to this message? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q13: How likely are you to believe a WEA message received on your mobile phone if you have 
not previously been made aware of the WEA program via other means (e.g., newspaper articles, 
mobile service provider mailings, and TV and radio news stories)? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q14: At 8:05 PM, while in the town of Kent, your cell phone receives and displays the following 
WEA message “Gunfire in this area Take shelter now. Kent PD” How likely are you to view this 
message as relevant to you? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 
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Q15: At 8:05 PM, your cell phone receives and displays the following WEA message: “Severe 
Hailstorm Warning in this area from 08:35 until 9:30 PM. Take shelter now.” How likely are you to 
take action in response to this message? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q16: How likely are you to opt out of the WEA service if you can confirm WEA message infor¬ 
mation via other channels such as radio or TV news? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q17: How likely are you to opt out of the WEA service if you view the WEA messages that you 
receive as spam? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q18: How likely are you to opt out of the WEA service if you receive multiple WEA messages 
from different alert originators (e.g., local emergency manager, county emergency manager, 
state emergency manager) that are not coordinated, or do not agree? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 
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Q19: How likely are you to believe a WEA message if you typically receive them twice each 
month? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q20: How likely are you to view a WEA message as spam if you typically receive them twice a 
week? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q21: At 8:05 PM, your cell phone receives and displays the following WEA message: “Nuclear 
Power Plant Warning in this area Take shelter now.” How likely are you to view this message as 
relevant to you? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q22: At 8:05 PM, your cell phone receives and displays the following WEA message: “Nuclear 
Power Plant Warning in this area Take shelter now to avoid radiation exposure.” How likely are 
you to believe this message? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 
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Q23: At 8:05 PM, your cell phone receives and displays a WEA message from the President of 
the United States describing an emergency event of national importance. How likely are you to 
view this message as spam? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q24: That concludes the WEA survey. Thank you again for your time in helping us to improve 
the WEA deployment! Please advance forward one last screen for the final capture of your 
answers. 

Note: Q25 is presented only if the respondent answers “No” to any question in Q3. 

Q25: We’re sorry that you’ve chosen not to participate in our survey. Thank you for your time! 
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Public Survey 2 

Q1: A Survey for the Wireless Emergency Alert Service 
WEA Background 

The Wireless Emergency Alerts (WEA) service is a partnership between the FCC, FEMA, and 
wireless carriers, whose purpose is to enhance public safety. WEA enables authorized federal, 
state, and local government authorities to send geographically targeted, text-like alerts to the pub¬ 
lic via wireless mobile devices such as cell phones and tablets. This service can deliver alerts from 
the president of the United States, AMBER alerts, and alerts regarding imminent local threats 
such as floods, chemical spills, civil unrest, etc., to all compatible mobile devices within the geo¬ 
graphic area specified by the originator of the alert. 

On compatible mobile devices, WEA uses a unique signal and vibration to attract attention. WEA 
messages will not have to be opened as do SMS text messages, but will pop up on the device’s 
screen. The public does not need to sign up to receive WEA messages — participating cell carriers 
will sell WEA-capable phones with the service already opted-in, and there is no charge for the 
WEA service or the messages you receive. 

This Survey 

The purpose of this survey is to help government authorities use WEA in a manner that enhances 
public safety. For WEA to be effective, alert originators must be able to issue alerts that are re¬ 
ceived, understood, believed, and acted upon by the public. This survey explores several factors 
that influence public response to alerts. 

Q2: Online Consent 

This survey is part of a research study conducted by Robert W. Stoddard II at Carnegie Mellon 
University. The purpose of the research is to study the factors that may influence the public’s or 
alert originator’s trust in the soon-to-be-deployed United States Wireless Emergency Alert (WEA) 
notification system. 

Procedures 

Survey recipients will be asked approximately 20 multiple choice questions regarding their reac¬ 
tions to different aspects and content of potential alert messages. The survey is expected to take 
no more than 5-7 minutes. 

Participant Requirements 

Participation in this study is limited to individuals age 18 and older. Participants must currently 
use some form of a cellular phone. 

Risks 

The discomfort and risk associated with participation in this study are no greater than those ordi¬ 
narily encountered in daily life or during other online activities. Discomfort may arise from the 
need to maintain focus and attention when responding to the 20 multiple choice questions that 
include repetition and subtle differences within the message content. The risk of a breach of con¬ 
fidentiality resulting in the inadvertent release of email addresses will be managed by the re- 
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searcher by ensuring secure access to this information through password control of the online sur¬ 
vey tool, subsequent removal of the information from the online system, and archival storage in a 
secured filing cabinet under control of the researcher. 

Benefits 

There may be no personal benefit from your participation in the study but the knowledge received 
may be of value to humanity. Respondents will be notified of a future, free, invitation-only webi¬ 
nar providing more detail of the Wireless Emergency Alert system. 

Compensation and Costs 

There is no compensation for participation in this study. There will be no cost to you if you partic¬ 
ipate in this study. 

Confidentiality 

By participating in this research, you understand and agree that Carnegie Mellon may be required 
to disclose your consent form, data, and other personally identifiable information as required by 
law, regulation, subpoena, or court order. Otherwise, your confidentiality will be maintained in 
the following manner: Your data and consent form will be kept separate. Your consent form will 
be stored in a locked location on Carnegie Mellon property and will not be disclosed to third par¬ 
ties. By participating, you understand and agree that the data and information gathered during this 
study may be used by Carnegie Mellon and published and/or disclosed by Carnegie Mellon to 
others outside of Carnegie Mellon. However, your name, address, contact information and other 
direct personal identifiers in your consent form will not be mentioned in any such publication or 
dissemination of the research data and/or results by Carnegie Mellon. 

Right to Ask Questions and Contact Information 

If you have any questions about this study, you should feel free to ask them by contacting the 
Principal Investigator now at Robert W. Stoddard If, Principal Researcher, Software Engineering 
Institute, Carnegie Mellon University, 4500 Fifth Avenue (Office 3110), Pittsburgh, PA 15213, 
412-268-1121, rws@sei.cmu.edu. If you have questions later, desire additional information, or 
wish to withdraw your participation please contact the Principle Investigator by mail, phone, or e- 
mail in accordance with the contact information listed above. 

If you have queslions perlaining to your rights as a research participant; or to report objections to 
this study, you should contact the Research Regulatory Compliance Office at Carnegie Mellon 
University. Email: irb-review@andrew.cmu.edu. Phone: 412-268-1901 or 412-268-5460. 

Voluntary Participation 

Your participation in this research is voluntary. You may discontinue participation at any time 
during the research activity. 


CMU/SEI-2013-SR-021 | 102 


Q3: Please answer the following three questions related to your consent. If you answer “No” to 


any question, you will be ineligible to take the survey. 



Yes (1) 

No (2) 

Are you age 18 or older? (1) 

o 

o 

Have you read and do you understand the previous online consent information? (2) 

o 

o 

Do you want to participate in this research and continue with the survey? (3) 

o 

o 


NOTE: Q4 through Q25 are presented only if the respondent answers “Yes” 

to all questions in Q3. 


Q4: Instructions 


Each of the following survey questions presents you with a situation, and asks you to estimate the 
likelihood of a particular response to that situation. The choices for your answer are 
Definitely Not I would respond to the situation less than 5% of the time 
Very Probably Not I would respond to the situation 5-20% of the time 
Probably Not I would respond to the situation 20^0% of the time 

Undecided I would respond to the situation 40-60% of the time 

Probably I would respond to the situation 60-80% of the time 

Very Probably I would respond to the situation 80-95% of the time 

Definitely 1 would respond to the situation more than 95% of the time 


Q5: How likely are you to take action in response to a WEA message if you determine that the 
message is not relevant to you (e.g., it does not apply to your location, it is not issued in a timely 
manner, or it does not address an emergency that will affect you)? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 
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Q6 At 8:05 PM, while in the town of Kent, your cell phone receives and displays the following 
WEA message: “Hazardous Materials Warning in this area until 08:49 PM. Take shelter now. 

Kent Fire Dept.” How likely are you to take action in response to this message? 

o Definitely Not (1) 

o Very Probably Not (2) 

o Probably Not (3) 
o Undecided (4) 

o Probably (5) 

o Very Probably (6) 

o Definitely (7) 

Q7: At 8:05 PM, your cell phone receives and displays the following WEA message: “Fire 
Warning in this area until 08:49 PM. Evacuate now State Police.” How likely are you to believe 
this message? 

o Definitely Not (1) 

o Very Probably Not (2) 

o Probably Not (3) 
o Undecided (4) 

o Probably (5) 

o Very Probably (6) 

o Definitely (7) 

Q8: How likely are you to view a WEA message as scam if you have received prior WEA alerts 
that have not been relevant to you (e.g., they did not apply to your location, they were not issued 
in a timely manner, or they did not address an emergency that affected you)? 

o Definitely Not (1) 

o Very Probably Not (2) 

o Probably Not (3) 
o Undecided (4) 

o Probably (5) 

o Very Probably (6) 

o Definitely (7) 
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Q9: At 8:05 PM, your cell phone receives and displays the following WEA message: Hazerdus 
Mateirals Warning in this area until 08:49 PM. Take shelter now.” How likely are you to under¬ 
stand this message? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q10: You are in ZIP code 12345 in the town of Kent when your cell phone receives and displays 
the following WEA message: “Hazardous Materials Warning in ZIP 12330 and 12345 until 08:49 
PM. Take shelter now. Kent Fire Dept.” How likely are you to view this message as relevant to 
you? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q11: You are in ZIP code 12345 in the town of Kent when your cell phone receives and displays 
the following WEA message: “Hazardous Materials Warning in this area until 08:49 PM. Take 
shelter now. Kent Fire Dept.” How likely are you to view this message as relevant to you? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 
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Q12: At 8:05 PM, while in the town of Kent, your cell phone receives and displays the following 
WEA message: “Hazardous Materials Warning in this area Take shelter now. Kent Fire Dept.” 
How likely are you to take action in response to this message? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q13: How likely are you to view a WEA message received on your mobile phone as spam if you 
have not previously been made aware of the WEA program via other means (e.g., newspaper 
articles, mobile service provider mailings, and TV and radio news stories)? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q14: At 8:05 PM, while in the town of Kent, your cell phone receives and displays the following 
WEA message: “Law Enforcment Warning in this area. Take shelter now. Kent PD” How likely 
are you to believe this message? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 
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Q15: At 8:05 PM, your cell phone receives and displays the following WEA message: “Severe 
Hailstorm Warning in this area from 09:35 until 10:30 PM. Take shelter now.” How likely are you 
to take action in response to this message? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q16: How likely are you to believe WEA message that you receive if you cannot find confirma¬ 
tion of the information on social media (e.g., Twitter, Facebook)? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q17: How likely are you to view a WEA message as spam if you cannot confirm message infor¬ 
mation via other channels such as radio or TV news? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q18: How likely are you to believe a WEA service if you receive multiple WEA alerts from differ¬ 
ent alert originators (e.g., local emergency manager, county emergency manager, state emer¬ 
gency manager) that are not coordinated, or do not agree? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 


CMU/SEI-2013-SR-021 


107 


Q19: How likely are you to believe a WEA message if you typically receive them twice each day? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q20: How likely are you to believe a WEA message as spam if you typically receive them twice 
each year? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q21: How likely are you to view a WEA message as spam if you typically receive them twice a 
month? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q22: At 8:05 PM, your cell phone receives and displays the following WEA message: “Nuclear 
Power Plant Warning in this area Take shelter now to avoid radiation exposure” How likely are 
you to view this message as relevant to you? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 
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Q23: How likely are you to understand a WEA message that you receive if it is issued in a lan¬ 
guage that you understand, but one that is not your primary language, (i.e., The WEA message 
is issued in English. Your primary language is Spanish, but you also speak English)? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q24: At 8:05 PM, your cell phone receives and displays a WEA message from a state emergency 
management agency regarding an emergency in your area such as a weather event, a chemical 
spill, or a wild fire. How likely are you to view this alert as spam? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q25: That concludes the WEA survey. Thank you again for your time in helping us to improve 
the WEA deployment! Please advance forward one last screen for the final capture of your 
answers. 

Note: Q26 is presented only if the respondent answers “No” to any question in Q3. 

Q26: We’re sorry that you’ve chosen not to participate in our survey. Thank you for your time! 
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Public Survey 3 

Q1: A Survey for the Wireless Emergency Alert Service 
WEA Background 

The Wireless Emergency Alerts (WEA) service is a partnership between the FCC, FEMA, and 
wireless carriers, whose purpose is to enhance public safety. WEA enables authorized federal, 
state, and local government authorities to send geographically targeted, text-like alerts to the pub¬ 
lic via wireless mobile devices such as cell phones and tablets. This service can deliver alerts from 
the president of the United States, AMBER alerts, and alerts regarding imminent local threats 
such as floods, chemical spills, civil unrest, etc., to all compatible mobile devices within the geo¬ 
graphic area specified by the originator of the alert. 

On compatible mobile devices, WEA uses a unique signal and vibration to attract attention. WEA 
messages will not have to be opened as do SMS text messages, but will pop up on the device’s 
screen. The public does not need to sign up to receive WEA messages — participating cell carriers 
will sell WEA-capable phones with the service already opted-in, and there is no charge for the 
WEA service or the messages you receive. 

This Survey 

The purpose of this survey is to help government authorities use WEA in a manner that enhances 
public safety. For WEA to be effective, alert originators must be able to issue alerts that are re¬ 
ceived, understood, believed, and acted upon by the public. This survey explores several factors 
that influence public response to alerts. 

Q2: Online Consent 

This survey is part of a research study conducted by Robert W. Stoddard II at Carnegie Mellon 
University. The purpose of the research is to study the factors that may influence the public’s or 
alert originator’s trust in the soon-to-be-deployed United States Wireless Emergency Alert (WEA) 
notification system. 

Procedures 

Survey recipients will be asked approximately 20 multiple choice questions regarding their reac¬ 
tions to different aspects and content of potential alert messages. The survey is expected to take 
no more than 5-7 minutes. 

Participant Requirements 

Participation in this study is limited to individuals age 18 and older. Participants must currently 
use some form of a cellular phone. 

Risks 

The discomfort and risk associated with participation in this study are no greater than those ordi¬ 
narily encountered in daily life or during other online activities. Discomfort may arise from the 
need to maintain focus and attention when responding to the 20 multiple choice questions that 
include repetition and subtle differences within the message content. The risk of a breach of con¬ 
fidentiality resulting in the inadvertent release of email addresses will be managed by the re- 
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searcher by ensuring secure access to this information through password control of the online sur¬ 
vey tool, subsequent removal of the information from the online system, and archival storage in a 
secured filing cabinet under control of the researcher. 

Benefits 

There may be no personal benefit from your participation in the study but the knowledge received 
may be of value to humanity. Respondents will be notified of a future, free, invitation-only webi¬ 
nar providing more detail of the Wireless Emergency Alert system. 

Compensation and Costs 

There is no compensation for participation in this study. There will be no cost to you if you partic¬ 
ipate in this study. 

Confidentiality 

By participating in this research, you understand and agree that Carnegie Mellon may be required 
to disclose your consent form, data, and other personally identifiable information as required by 
law, regulation, subpoena, or court order. Otherwise, your confidentiality will be maintained in 
the following manner: Your data and consent form will be kept separate. Your consent form will 
be stored in a locked location on Carnegie Mellon property and will not be disclosed to third par¬ 
ties. By participating, you understand and agree that the data and information gathered during this 
study may be used by Carnegie Mellon and published and/or disclosed by Carnegie Mellon to 
others outside of Carnegie Mellon. However, your name, address, contact information and other 
direct personal identifiers in your consent form will not be mentioned in any such publication or 
dissemination of the research data and/or results by Carnegie Mellon. 

Right to Ask Questions and Contact Information 

If you have any questions about this study, you should feel free to ask them by contacting the 
Principal Investigator now at Robert W. Stoddard If, Principal Researcher, Software Engineering 
Institute, Carnegie Mellon University, 4500 Fifth Avenue (Office 3110), Pittsburgh, PA 15213, 
412-268-1121, rws@sei.cmu.edu. If you have questions later, desire additional information, or 
wish to withdraw your participation please contact the Principle Investigator by mail, phone, or e- 
mail in accordance with the contact information listed above. 

If you have queslions perlaining to your rights as a research participant; or to report objections to 
this study, you should contact the Research Regulatory Compliance Office at Carnegie Mellon 
University. Email: irb-review@andrew.cmu.edu. Phone: 412-268-1901 or 412-268-5460. 

Voluntary Participation 

Your participation in this research is voluntary. You may discontinue participation at any time 
during the research activity. 
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Q3: Please answer the following three questions related to your consent. If you answer “No” to 


any question, you will be ineligible to take the survey. 



Yes (1) 

No (2) 

Are you age 18 or older? (1) 

o 

o 

Have you read and do you understand the previous online consent information? (2) 

o 

o 

Do you want to participate in this research and continue with the survey? (3) 

o 

o 


NOTE: Q4 through Q25 are presented only if the respondent answers “Yes” 

to all questions in Q3. 


Q4: Instructions 

Each of the following survey questions presents you with a situation, and asks you to estimate the 
likelihood of a particular response to that situation. The choices for your answer are 
Definitely Not I would respond to the situation less than 5% of the time 

Very Probably Not I would respond to the situation 5-20% of the time 
Probably Not I would respond to the situation 20^0% of the time 

Undecided I would respond to the situation 40-60% of the time 

Probably I would respond to the situation 60-80% of the time 

Very Probably I would respond to the situation 80-95% of the time 

Definitely I would respond to the situation more than 95% of the time 

Q5: How likely are you to view a WEA message as spam if you determine that the message is 
not relevant to you (e.g., it does not apply to your location, it is not issued in a timely manner, or 
it does not address an emergency that affects you)? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 
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Q6 At 8:05 PM, while in the town of Kent, your cell phone receives and displays the following 
WEA message: “Hazardous Materials Warning in this area until 08:49 PM. Take shelter now. 

Kent Fire Dept.” How likely are you to take action in response to this message? 

o Definitely Not (1) 

o Very Probably Not (2) 

o Probably Not (3) 
o Undecided (4) 

o Probably (5) 

o Very Probably (6) 

o Definitely (7) 

Q7: At 8:05 PM, your cell phone receives and displays the following WEA message: “Fire 
Warning in this area until 08:49 PM. Evacuate now State Police.” How likely are you to believe 
this message? 

o Definitely Not (1) 

o Very Probably Not (2) 

o Probably Not (3) 
o Undecided (4) 

o Probably (5) 

o Very Probably (6) 

o Definitely (7) 

Q8: How likely are you to view a WEA message as scam if you have received prior WEA alerts 
that have not been relevant to you (e.g., they did not apply to your location, they were not issued 
in a timely manner, or they did not address an emergency that affected you)? 

o Definitely Not (1) 

o Very Probably Not (2) 

o Probably Not (3) 
o Undecided (4) 

o Probably (5) 

o Very Probably (6) 

o Definitely (7) 
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Q9: At 8:05 PM, your cell phone receives and displays the following WEA message: “Hazerdus 
Mateirals Warning in this area until 08:49 PM. Take shelter now.” How likely are you to under¬ 
stand this message? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q10: You are in ZIP code 12345 in the town of Kent when your cell phone receives and displays 
the following WEA message: “Hazardous Materials Warning in ZIP 12330 and 12345 until 08:49 
PM. Take shelter now. Kent Fire Dept.” How likely are you to view this message as relevant to 
you? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q11: You are in ZIP code 12345 in the town of Kent when your cell phone receives and displays 
the following WEA message: “Hazardous Materials Warning in this area until 08:49 PM. Take 
shelter now. Kent Fire Dept.” How likely are you to view this message as relevant to you? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 
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Q12: At 8:05 PM, while in the town of Kent, your cell phone receives and displays the following 
WEA message: “Hazardous Materials Warning in this area Take shelter now. Kent Fire Dept.” 
How likely are you to take action in response to this message? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q13: How likely are you to view a WEA message received on your mobile phone as spam if you 
have not previously been made aware of the WEA program via other means (e.g., newspaper 
articles, mobile service provider mailings, and TV and radio news stories)? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q14: At 8:05 PM, while in the town of Kent, your cell phone receives and displays the following 
WEA message: “Law Enforcment Warning in this area. Take shelter now. Kent PD” How likely 
are you to believe this message? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 
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Q15: At 8:05 PM, your cell phone receives and displays the following WEA message: “Severe 
Hailstorm Warning in this area from 09:35 until 10:30 PM. Take shelter now.” How likely are you 
to take action in response to this message? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q16: How likely are you to believe WEA message that you receive if you cannot find confirma¬ 
tion of the information on social media (e.g., Twitter, Facebook)? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q17: How likely are you to view a WEA message as spam if you cannot confirm message infor¬ 
mation via other channels such as radio or TV news? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q18: How likely are you to believe a WEA service if you receive multiple WEA alerts from differ¬ 
ent alert originators (e.g., local emergency manager, county emergency manager, state emer¬ 
gency manager) that are not coordinated, or do not agree? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 
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Q19: How likely are you to believe a WEA message if you typically receive them twice each day? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q20: How likely are you to believe a WEA message as spam if you typically receive them twice 
each year? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q21: How likely are you to view a WEA message as spam if you typically receive them twice a 
month? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q22: At 8:05 PM, your cell phone receives and displays the following WEA message: “Nuclear 
Power Plant Warning in this area Take shelter now to avoid radiation exposure.” How likely are 
you to view this message as relevant to you? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 
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Q23: How likely are you to understand a WEA message that you receive if it is issued in a lan¬ 
guage that you understand, but one that is not your primary language, (i.e., The WEA message 
is issued in English. Your primary language is Spanish, but you also speak English)? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q24: At 8:05 PM, your cell phone receives and displays a WEA message from a state emergency 
management agency regarding an emergency in your area such as a weather event, a chemical 
spill, or a wild fire. How likely are you to view this alert as spam? 

o Definitely Not (1) 
o Very Probably Not (2) 
o Probably Not (3) 
o Undecided (4) 
o Probably (5) 
o Very Probably (6) 
o Definitely (7) 

Q25: That concludes the WEA survey. Thank you again for your time in helping us to improve 
the WEA deployment! Please advance forward one last screen for the final capture of your 
answers. 

Note: Q26 is presented only if the respondent answers “No” to any question in Q3. 

Q26: We’re sorry that you’ve chosen not to participate in our survey. Thank you for your time! 
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Appendix K Mapping of Pubiic Trust Modei Reiationships to Survey Questions 


Factor Relationship 

Analysis 

# 

Qualtrics 

Question 

23_Who should act => 1_Relevance 

delta 

15 

2.06 

s2q10 

You are in ZIP code 12345 in the town of Kent when your cell 
phone receives and displays the following WEA alert 
“Hazardous Materials Warning 
in ZIP 12330 and 12345 
until 08:49 PM. 

Take shelter now. 

Kent Fire Dept.” 

How likely are you to view this alert as relevant to you? 

16 

3.06 

s3q10 

You are in ZIP code 12345 in the town of Kent when your cell 
phone receives and displays the following WEA alert 
“Hazardous Materials Warning 
in this area 
until 08:49 PM. 

Take shelter now. 

Kent Fire Dept.” 

How likely are you to view this alert as relevant to you? 

17 

1.07 

s1q11 

You are in ZIP code 12345 in the town of Kent when your cell 
phone receives and displays the following WEA alert 
“Hazardous Materials Warning 
until 08:49 PM. 

Take shelter now. 

Kent Fire Dept.” 

How likely are you to view this alert as relevant to you? 
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Factor Relationship 

Analysis 

# 

Qualtrics 

Question 

30_ Explain what (has happened) => 1_Relevance 

delta 

11 

3.09 

s3q13 

At 8:05 PM, while in the town of Kent, your cell 
phone receives and displays the following WEA alert 
“Law Enforcement Warning 
in this area 

Take shelter now. 

Kent PD” 

How likely are you to view this alert as relevant to you? 

12 

1.1 

s1q14 

At 8:05 PM, while in the town of Kent, your cell phone 
receives and displays the following WEA alert 
“Gunfire 
in this area 

Take shelter now. 

Kent PD” 

How likely are you to view this alert as relevant to you? 

70_Why 1 should act => 1_Relevance 

delta 

13 

1.18 

s1q21 

At 8:05 PM, your cell phone receives and displays 
the following WEA alert 
“Nuclear Power Plant Warning 
in this area 

Take shelter now” 

How likely are you to view this alert as relevant to you? 

14 

2.18 

s2q22 

At 8:05 PM, your cell phone receives and displays 
the following WEA alert 
“Nuclear Power Plant Warning 
in this area 

Take shelter now to avoid radiation exposure” 

How likely are you to view this alert as relevant to you? 

1_Relevance => 100_Believing 

direct 

25 

1.01 

s1q5 

How likely are you to believe a WEA Alert if you 
determine that the alert is not relevant to you 
(e.g., it does not apply to your location, it is not issued 
in a timely manner, it does not address an emergency 
that will affect you)? 
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Factor Relationship 

Analysis 

# 

Qualtrics 

Question 

12_Alert source => 100_Believing 

delta 

29 

3.02 

s3q6 

At 8:05 PM, your cell phone receives and displays the 
following WEA alert 
“Fire Warning 
in this area 
until 08:49 PM. 

Evacuate now 

Local PD” 

How likely are you to believe this alert? 

30 

1.03 

s1q7 

At 8:05 PM, your cell phone receives and displays the 
following WEA alert 
“Fire Warning 
in this area 
until 08:49 PM. 

Evacuate now 

State Police” 

How likely are you to believe this alert? 

31 

2.03 

s2q7 

At 8:05 PM, your cell phone receives and displays the 
following WEA alert 
“Fire Warning 
in this area 
until 08:49 PM. 

Evacuate now 

FBI” 

How likely are you to believe this alert? 

20_History of relevance => 100_Believing 

direct 

24 

1.04 

s1q8 

How likely are you to believe a WEA alert if you have 
received prior WEA alerts that have not been relevant to you 
(e.g., they did not apply to your location, they were not issued 
in a timely manner, they did not address an emergency that 
affected you)? 
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Factor Relationship 

Analysis 

# 

Qualtrics 

Question 

21_Clarity of message spelling and grammar => 100_Believing 

delta 

20 

3.18 

s3q21 

At 8:05 PM, your cell phone receives and displays the follow¬ 
ing WEA alert 

“Hazerdus Mateirals Warning 
in this area 
until 08:49 PM. 

Take shelter now” 

How likely are you to believe this alert? 

36 

3.04 

s3q8 

At 8:05 PM, your cell phone receives and displays the follow¬ 
ing WEA alert 

“Hazardous Materials Warning 
in this area 
until 08:49 PM. 

Take shelter now” 

How likely are you to believe this alert? 

26_Where to go for more information => 100_Believing 

delta 

13 

1.18 

s1q21 

At 8:05 PM, your cell phone receives and displays the follow¬ 
ing WEA alert 

“Nuclear Power Plant Warning 
in this area 

Take shelter now” 

How likely are you to view this alert as relevant to you? 

38 

3.08 

s3q12 

At 8:05 PM, your cell phone receives and displays the follow¬ 
ing WEA alert 

“Nuclear Power Plant Warning 
in this area 

Take shelter now 

Check TV & Radio for more info” 

How likely are you to believe this alert? 

3_Public awareness of WEA => 100_Believing 

direct 

27 

1.09 

s1q13 

How likely are you to believe a WEA Alert received on your 
mobile phone if you have not previously been made aware of 
the WEA program via other means (e.g., newspaper articles, 
mobile service provider mailings, TV, and radio news stories)? 
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Factor Relationship 

Analysis 

# 

Qualtrics 

Question 

30_Explain what has happened 


100_Believing 

delta 

18 

2.1 

s2q14 

At 8:05 PM, while in the town of Kent, your cell phone receives 
and displays the following WEA alert 
“Law Enforcement Warning 
in this area 

Take shelter now. 

Kent PD” 

How likely are you to believe this alert? 

=> 

19 

3.1 

s3q14 

At 8:05 PM, while in the town of Kent, your cell phone receives 
and displays the following WEA alert 
“Gunfire 
in this area 

Take shelter now. 

Kent PD” 

How likely are you to believe this alert? 

37_Confirmation via social media 

=> 

100_Believing 

direct 

37 

2.12 

s2q16 

How likely are you to believe a WEA Alert that you receive if 
you cannot find confirmation of the information on social media 
(e.g., Twitter, Facebook)? 

44_Redundancy of alerting 

=> 

100_Believing 

direct 

22 

3.12 

s3q15 

How likely are you to believe a WEA Alert if you cannot con¬ 
firm the alert information via other channels such as radio or 

TV news? 

55_Local jurisdictions act uncoordinated 

=> 

100_Believing 

direct 

26 

2.14 

s2q18 

How likely are you to believe a WEA Alert if you receive multi¬ 
ple WEA alerts from different alert originators (e.g., local 
emergency manager, county emergency manager, state 
emergency manager) that are not coordinated, or do not 
agree? 





32 

2.15 

s2q19 

How likely are you to believe a WEA alert if you typically re¬ 
ceive them twice each day? 

7_Frequency 


100_Believing 

delta 

33 

3.15 

s3q18 

How likely are you to believe a WEA alert if you typically re¬ 
ceive them twice each week? 

=> 

34 

1.16 

s1q19 

How likely are you to believe a WEA alert if you typically re¬ 
ceive them twice each month? 





35 

2.16 

s2q20 

How likely are you to believe a WEA alert if you typically re¬ 
ceive them twice each year? 
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Factor Relationship 

Analysis 

# 

Qualtrics 

Question 

70_Explain why I should act ^ 100_Believing 

delta 

13 

1.18 

s1q21) 

At 8:05 PM, your cell phone receives and displays the follow¬ 
ing WEA alert 

“Nuclear Power Plant Warning 
in this area 

Take shelter now” 

How likely are you to view this alert as relevant to you? 

21 

1.19 

s1q22 

At 8:05 PM, your cell phone receives and displays the follow¬ 
ing WEA alert 

“Nuclear Power Plant Warning 
in this area 

Take shelter now to avoid radiation exposure” 

How likely are you to believe this alert? 

8_History of final communication ^ 100_Believing 

direct 

28 

3.19 

s3q22 

How likely are you to believe a WEA Alert if prior WEA alerts 
that you have received have not included an “all clear” mes¬ 
sage after the event has been resolved? 

48_ Alerts viewed as spam ^ 101_Hearing 

direct 

61 

3.13 

s3q16 

How likely are you to read a WEA alert that you receive if you 
view WEA alerts as spam? 

1_Relevance ^ 103_Acting 

direct 

10 

2.01 

s2q5 

How likely are you to take action in response to a WEA Alert if 
you determine that the alert is not relevant to you (e.g., it does 
not apply to your location, it is not issued in a timely manner, it 
does not address an emergency that will affect you)? 

10_Action to take ^ 103_Acting 

delta 

2 

1.02 

s1q6 

At 8:05 PM, while in the town of Kent, your cell phone receives 
and displays the following WEA alert 
“Hazardous Materials Warning 
in this area 
until 08:49 PM. 

Take shelter now. 

Kent Fire Dept.” 

How likely are you to take action in response to this alert? 

3 

2.02 

s2q6 

At 8:05 PM, while in the town of Kent, your cell phone receives 
and displays the following WEA alert 
“Hazardous Materials Warning 
in this area 
until 08:49 PM. 

Kent Fire Dept.” 

How likely are you to take action in response to this alert? 
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Factor Relationship 

Analysis 

# 

Qualtrics 

Question 




7 

2.07 

s2q11 

You are in ZIP code 12345 in the town of Kent when your cell 
phone receives and displays the following WEA alert 
“Hazardous Materials Warning 
in this area 
until 08:49 PM. 

Take shelter now. 

Kent Fire Dept.” 

How likely are you to take action in response to this alert? 

23_Who should act 

^ 103_Acting 

delta 

8 

3.07 

s3q11 

You are in ZIP code 12345 in the town of Kent when your cell 
phone receives and displays the following WEA alert 
“Hazardous Materials Warning 
in ZIP 12330 and 12345 
until 08:49 PM. 

Take shelter now. 

Kent Fire Dept.” 

How likely are you to take action in response to this alert? 




9 

1.08 

s1q12 

You are in ZIP code 12345 in the town of Kent when your cell 
phone receives and displays the following WEA alert 
“Hazardous Materials Warning 
until 08:49 PM. 

Take shelter now. 

Kent Fire Dept.” 

How likely are you to take action in response to this alert? 

24 Time window to act 

103 Acting 

delta 

2 

1.02 

s1q6 

At 8:05 PM, while in the town of Kent, your cell phone receives 
and displays the following WEA alert 
“Hazardous Materials Warning 
in this area 
until 08:49 PM. 

Take shelter now. 

Kent Fire Dept.” 

How likely are you to take action in response to this alert? 




4 

2.08 

s2q12 

At 8:05 PM, while in the town of Kent, your cell phone receives 
and displays the following WEA alert 
“Hazardous Materials Warning 
in this area 

Take shelter now. 

Kent Fire Dept.” 

How likely are you to take action in response to this alert? 
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Factor Relationship 

Analysis 

# 

Qualtrics 

Question 

32_Lead time provided => 103_Acting 

delta 

5 

1.11 

s1q15 

At 8:05 PM, your cell phone receives and displays the follow¬ 
ing WEA alert 
“Severe Hailstorm Warning 
in this area 

from 08:35 until 9:30 PM. 

Take shelter now” 

How likely are you to take action in response to this alert? 

6 

2.11 

s2q15 

At 8:05 PM, your cell phone receives and displays the follow¬ 
ing WEA alert 
“Severe Hailstorm Warning 
in this area 

from 09:35 until 10:30 PM. 

Take shelter now 

How likely are you to take action in response to this alert? 

15_Easy additional follow us mechanisms => 103_Understanding 

direct 

43 

3.03 

s3q7 

How likely are you to understand a WEA alert that you receive 
if the sender (e.g. local emergency manager, county emer¬ 
gency manager, state emergency manager) does not have 
alternate communications channels (e.g., web site, Facebook 
page, Twitter account) that you can access easily. 

21_Clarity of message spelling and grammar => 103_Understanding 

delta 

40 

1.05 

s1q9 

At 8:05 PM, your cell phone receives and displays the follow¬ 
ing WEA alert 

“Hazerdus Mateirals Warning 
in this area 
until 08:49 PM. 

Take shelter now” 

How likely are you to understand this alert? 

41 

2.05 

s2q9 

At 8:05 PM, your cell phone receives and displays the follow¬ 
ing WEA alert 

“Hazardous Materials Warning 
in this area 
until 08:49 PM. 

Take shelter now” 

How likely are you to understand this alert? 

55_Local jurisdictions act uncoordinated => 103_Understanding 

direct 

39 

3.14 

s3q17 

How likely are you to understand a WEA Alert if you receive 
multiple WEA alerts from different alert originators (e.g. local 
emergency manager, county emergency manager, state 
emergency manager) that are not coordinated, or do not 
agree? 
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Factor Relationship 

Analysis 

# 

Qualtrics 

Question 

71_Message in primary language => 103_Understanding 

direct 

42 

2.19 

s2q23 

How likely are you to understand a WEA Alert that you receive 
if it is issued in a language that you understand, but one that is 
not your primary language, (i.e., The WEA alert is issued in 
English. Your primary language is Spanish, but you also speak 
English.) 

44_Redundancy of alerting => 4_Opt out rate 

direct 

45 

1.13 

s1q16 

How likely are you to opt out of the WEA service if you can 
confirm WEA alert information via other channels such as 
radio or TV news? 

48_ Alerts viewed as spam => 4_Opt out rate 

direct 

46 

1.14 

s1q17 

How likely are you to opt out of the WEA service if you view 
the WEA alerts that you receive as spam. 

55_Local jurisdictions act uncoordinated => 4_Opt out rate 

direct 

44 

1.15 

s1q18 

How likely are you to opt out of the WEA service if you receive 
multiple WEA alerts from different alert originators (e.g. local 
emergency manager, county emergency manager, state 
emergency manager) that are not coordinated, or do not 
agree. 

1_Relevance =*• 48_ Alerts viewed as spam 

direct 

56 

3.01 

s3q5 

How likely are you to view a WEA Alert as spam if you deter¬ 
mine that the alert is not relevant to you (e.g., it does not apply 
to your location, it is not issued in a timely manner, it does not 
address an emergency that affects you). 

20_History of relevance => 48_ Alerts viewed as spam 

direct 

55 

2.04 

s2q8 

How likely are you to view a WEA Alert as spam if you have 
received prior WEA alerts that have not been relevant to you 
(e.g., they did not apply to your location, they were not issued 
in a timely manner, they did not address an emergency that 
affected you)? 
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Factor Relationship 

Analysis 

# 

Qualtrics 

Question 

21_Clarity of message spelling and grammar => 48_ Alerts viewed as spam 

delta 

57 

3.05 

s3q9 

At 8:05 PM, your cell phone receives and displays the follow¬ 
ing WEA alert 

“Hazerdus Mateirals Warning 
in this area 
until 08:49 PM. 

Take shelter now” 

How likely are you to view this alert as spam? 

58 

1.06 

s1q10 

At 8:05 PM, your cell phone receives and displays the follow¬ 
ing WEA alert 

“Hazardous Materials Warning 
in this area 
until 08:49 PM. 

Take shelter now” 

How likely are you to view this alert as spam? 

3_Public awareness of WEA => 48_ Alerts viewed as spam 

direct 

54 

2.09 

s2q13 

How likely are you to view a WEA Alert received on your mo¬ 
bile phone as spam if you have previously been made aware 
of the WEA program via other means (e.g., newspaper arti¬ 
cles, mobile service provider mailings, and TV and radio news 
stories). 

44_Redundancy of alerting => 48_ Alerts viewed as spam 

direct 

59 

2.13 

s2q17 

How likely are you to view a WEA message as spam if you 
cannot confirm alert information via other channels such as 
radio or TV news? 

7_Frequency => 48_ Alerts viewed as spam 

delta 

50 

3.16 

s3q19 

How likely are you to view a WEA alert as spam if your typical¬ 
ly receive them twice a day 

51 

1.17 

s1q20 

How likely are you to view a WEA alert as spam if your typical¬ 
ly receive them twice a week 

52 

2.17 

s2a21 

How likely are you to view a WEA alert as spam if your typical¬ 
ly receive them twice a month 

53 

3.17 

s3q20 

How likely are you to view a WEA alert as spam if your typical¬ 
ly receive them twice a year 
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Factor Relationship 

Analysis 

# 

Qualtrics 

Question 




47 

1.2 

s1q23 

At 8:05 PM, your cell phone receives and displays a WEA alert 
from the President of the United States describing an emer¬ 
gency event of national importance. 

How likely are you to view this alert as spam? 

99_Type of alert 

=> 48_ Alerts viewed as spam 

delta 

48 

2.2 

s2q24 

At 8:05 PM, your cell phone receives and displays a WEA alert 
from a state emergency management agency regarding an 
emergency in your area such as a weather event, a chemical 
spill, or a wildfire? How likely are you to view this alert as 
spam? 




49 

3.2 

s3q23 

At 8:05 PM, your state emergency management agency is¬ 
sues an AMBER (America’s Missing: Broadcast Emergency 
Response) alert regarding a child abduction in your area. Your 
cell phone receives and displays this information as a WEA 
alert. 

How likely are you to view this alert as spam? 
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Appendix L Alert Originator Surveys 


Alert Originator Survey 1 

Q1: A Survey for the Wireless Emergency Alert Service 

Dear Public Safety Colleague: 

In support of the First Responders Group of the U. S. Department of Flomeland Security, the 
Software Engineering Institute (SEI) of Carnegie Mellon University is performing research to aid 
Emergency Management Agencies (EMAs) in the adoption and use of the Wireless Emergency 
Alert (WEA) Service (formerly known as the Commercial Mobile Alert Service, or CMAS). If 
you are unfamiliar with WEA, please see the notes below. For WEA to be effective, the public 
must “trust” the alerts that they receive. Furthermore, if alert originators are to use the WEA ser¬ 
vice, they must “trust” it perform properly. To explore these issues of trust, the SEI is conducting 
surveys of the public, and surveys of the alert origination community. 

As a member of the public safety community, we ask you participate in this voluntary survey to 
assist us in our efforts to enhance public safety. Completion of the survey should require less than 
10 minutes. Participation is limited to individuals 18 years of age or older. As a reward for your 
participation, survey respondents will receive instructions to access a free, follow-up, invitation- 
only webinar sharing further details of the WEA system. 

Thank you for your assistance in this important study. 

Notes on the WEA Service 

WEA is a partnership between the FCC, FEMA and wireless carriers. WEA enables federal, state, 
and local government authorities to send geographically targeted, text-like alerts to the public via 
wireless mobile devices such as cell phones and tablets. This service can be used to deliver alerts 
from the President of the United States, AMBER alerts, and alerts regarding imminent local 
threats such as floods, chemical spills, civil unrest, etc. These alerts are sent to all compatible mo¬ 
bile devices within the geographic area specified by the alert originator. For more information 
regarding WEA, please see http://www.fema.gov/integrated-public-alert-waming-system. 

Q2: Online Consent 

This survey is part of a research study conducted by Robert W. Stoddard II at Carnegie Mellon 
University. The purpose of the research is to study the factors that may influence the public’s or 
alert originator’s tmst in the soon-to-be-deployed U.S. Wireless Emergency Alert (WEA) notifi¬ 
cation system. 

Procedures 

Survey recipients will be asked approximately 12 questions regarding their reactions to different 
aspects and content of potential alert messages. The survey is expected to take no more than 10 
minutes. 
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Participant Requirements 

Participation in this study is limited to individuals age 18 and older. Participants must currently 
use some form of a cellular phone. 

Risks 

The discomfort and risk associated with participation in this study are no greater than those ordi¬ 
narily encountered in daily life or during other online activities. Discomfort may arise from the 
need to maintain focus and attention when responding to the 20 multiple choice questions that 
include repetition and subtle differences within the message content. The risk of a breach of con¬ 
fidentiality resulting in the inadvertent release of email addresses will be managed by the re¬ 
searcher by ensuring secure access to this information through password control of the online 
survey tool, subsequent removal of the information from the online system, and archival storage 
in a secured filing cabinet under control of the researcher. 

Benefits 

There may be no personal benefit from your participation in the study but the knowledge received 
may be of value to humanity. Respondents will be notified of a future, free, invitation-only webi¬ 
nar providing more detail of the Wireless Emergency Alert system. 

Compensation & Costs 

There is no compensation for participation in this study. There will be no cost to you if you partic¬ 
ipate in this study. 

Confidentiality 

By participating in this research, you understand and agree that Carnegie Mellon may be required 
to disclose your consent form, data and other personally identifiable information as required by 
law, regulation, subpoena or court order. Otherwise, your confidentiality will be maintained in the 
following manner: Your data and consent form will be kept separate. Your consent form will be 
stored in a locked location on Carnegie Mellon property and will not be disclosed to third parties. 
By participating, you understand and agree that the data and information gathered during this 
study may be used by Carnegie Mellon and published and/or disclosed by Carnegie Mellon to 
others outside of Carnegie Mellon. However, your name, address, contact information and other 
direct personal identifiers in your consent form will not be mentioned in any such publication or 
dissemination of the research data and/or results by Carnegie Mellon. 

Right to Ask Questions & Contact Information 

If you have any questions about this study, you should feel free to ask them by contacting the 
Principal Investigator now at Robert W. Stoddard II, Principal Researcher, Software Engineering 
Institute, Carnegie Mellon University, 4500 Fifth Avenue (Office 3110), Pittsburgh, PA 15213, 
412-268-1121, rws@sei.cmu.edu]. If you have questions later, desire additional information, or 
wish to withdraw your participation please contact the Principle Investigator by mail, phone or e- 
mail in accordance with the contact information listed above. 
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If you have questions pertaining to your rights as a research participant; or to report objections to 
this study, you should contact the Research Regulatory Compliance Office at Carnegie Mellon 
University. Email: irb-review@andrew.cmu.edu . Phone: 412-268-1901 or 412-268-5460. 

Voluntary Participation 

Your participation in this research is voluntary. You may discontinue participation at any time 
during the research activity. 

Q3: Please answer the following three questions related to your consent. 

If you answer “No” to any question, you will be ineligible to take the survey. 



Yes 

No 

Are you age 18 or older? 

o 

o 

Have you read and do you understand the previous online consent information? 

o 

o 

Do you want to participate in this research and continue with the survey? 

o 

o 


NOTE: Q4 through Q16 are presented only if the respondent answers “Yes” to all questions in Q3. 


Q4: Instructions 

Each of the following survey questions presents you with a situation, and asks you to estimate the 
likelihood of a particular response to that situation. The choices for your answer are 
Definitely Not I would respond to the situation less than 5% of the time. 

Very Probably Not I would respond to the situation 5-20% of the time. 

Probably Not I would respond to the situation 20^0% of the time. 

Undecided I would respond to the situation 40-60% of the time. 

Probably I would respond to the situation 60-80% of the time. 

Very Probably I would respond to the situation 80-95% of the time. 

Definitely I would respond to the situation more than 95% of the time. 

Q5: What is the approximate population within your jurisdiction (in thousands)? 

Residents _ 

Transients _ 


Q6: Does your agency presently issue public alerts via email, Emergency Notification System, 
EAS, or other mechanism? 

o No 
o Yes 

o Don’t Know 
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Q7: Do you think WEA would be an appropriate tool to issue a public alert for an event that 
requires public action _ 



Definitely 

Not 

Very Proba¬ 
bly Not 

Probably 

Not 

Maybe 

Probably 

Very Prob¬ 
ably 

Definitely 

within 10 
minutes? 

o 

o 

o 

o 

o 

o 

o 

within 30 
minutes? 

o 

o 

o 

o 

o 

o 

o 

within 60 
minutes? 

o 

o 

o 

o 

o 

o 

o 

within 2 
hours? 

o 

o 

o 

o 

o 

o 

o 


Q8: Do you think WEA would be an appropriate tool to issue a public alert for a severe and 
urgent event where _ 


Definitely Very Probably Maybe Probably Very Definitely 

Not Probably Not Probably 

Not 


30% of the alert recip¬ 
ients are in the haz¬ 
ard zone and 70% 
are outside the zone? 

o 

o 

o 

o 

o 

o 

o 








50% of the alert recip¬ 
ients are in the haz¬ 
ard zone and 50% 
are outside the zone? 

o 

o 

o 

o 

o 

o 

o 








70% of the alert recip¬ 
ients are in the haz¬ 
ard zone and 30% 
are outside the zone? 

o 

o 

o 

o 

o 

o 

o 








90% of the alert recip¬ 
ients are in the haz¬ 
ard zone and 10% 
are outside the zone? 

o 

o 

o 

o 

o 

o 

o 
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Q9: Are you likely to use WEA to issue a public alert for a severe and urgent event if the WEA 
service is typically _ 



Definitely 

Not 

Very 

Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

unavailable for use 1 
hour per week (i.e., 
99.4% availability) for 
maintenance? 

o 

o 

o 

o 

o 

o 

o 

unavailable for use 1 
hour per month (i.e., 
99.9% availability) for 
maintenance? 

o 

o 

o 

o 

o 

o 

o 

unavailable for use 1 
hour per year (i.e., 
99.99% availability) for 
maintenance? 

o 

o 

o 

o 

o 

o 

o 


Q10: Are you likely to use WEA to issue a public alert for a severe and urgent event if the 
process of creating and issuing an alert _ 



Definitely 

Not 

Very Proba¬ 
bly Not 

Probably 

Not 

Maybe 

Probably 

Very Prob¬ 
ably 

Definitely 

takes 5 
minutes 

o 

o 

o 

o 

o 

o 

o 

takes 10 
minutes 

o 

o 

o 

o 

o 

o 

o 

takes 20 
minutes 

o 

o 

o 

o 

o 

o 

o 

takes 40 
minutes 

o 

o 

o 

o 

o 

o 

o 


Q11: Are you likely to use WEA to issue a public alert fora severe and urgent event if the 
creation and issuance of a WEA alert is accomplished using a system that is _ 



Definitely 

Not 

Very 

Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

independent of other 
alerting and/or emer¬ 
gency management 
systems in your office? 

o 

o 

o 

o 

o 

o 

o 

integrated with other 
alerting and/or emer¬ 
gency management 
systems in your office? 

o 

o 

o 

o 

o 

o 

o 
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Q12: Are you likely to use WEA to issue a public alert if you believe you have 



Definitely 

Not 

Very 

Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

a thorough under¬ 
standing of the princi¬ 
ples and applications 
of the WEA service 

O 

O 

o 

o 

o 

o 

o 

a moderate under¬ 
standing of the princi¬ 
ples and applications 
of the WEA service 

o 

o 

o 

o 

o 

o 

o 

a minimal understand¬ 
ing of the principles 
and applications of the 
WEA service 

o 

o 

o 

o 

o 

o 

o 


Q13: Would you be confident that the WEA service had delivered your alert if you received 



Definitely 

Not 

Very 

Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

no response from the 
WEA service regarding 
the status of your alert 
from IPAWS-OPEN? 

o 

o 

o 

o 

o 

o 

o 

a response from the 

WEA service indicating 
that your message had 
been received by 
IPAWS-OPEN? 

o 

o 

o 

o 

o 

o 

o 

a response from the 

WEA service indicating 
that your message had 
been received and ac¬ 
cepted by IPAWS- 
OPEN? 

o 

o 

o 

o 

o 

o 

o 

a response from the 

WEA service indicating 
that your message had 
been received and ac¬ 
cepted by IPAWS- 
OPEN, and sent to the 
wireless carriers? 

o 

o 

o 

o 

o 

o 

o 

a response from the 

WEA service indicating 
that your message had 
been received and ac¬ 
cepted by IPAWS- 
OPEN, sent to the wire¬ 
less carriers, and 
transmitted by the wire¬ 
less carriers? 

o 

o 

o 

o 

o 

o 

o 


CMU/SEI-2013-SR-021 | 135 
























Q14: Are you likely to use WEA to issue a public alert for a severe and urgent event if findings of 
internal After Action Reviews of prior alerts have been _ 



Definitely 

Not 

Very Proba¬ 
bly Not 

Probably 

Not 

Maybe 

Probably 

Very Prob¬ 
ably 

Definitely 

unfavorable? 

o 

o 

O 

o 

o 

o 

o 

neutral? 

o 

o 

o 

o 

o 

o 

o 

favorable? 

o 

o 

o 

o 

o 

o 

o 


Q15 Are you likely to use WEA to issue a public alert for a severe and urgent event if prior alerts 


have been disseminated ... 



Definitely 

Not 

Very Prob¬ 
ably Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

within 2 minutes of 








your alert data 
input? 

within 2 to 5 

o 

o 

o 

o 

o 

o 

o 

minutes of your 
alert data input? 

within 5 to 10 

o 

o 

o 

o 

o 

o 

o 

minutes of your 
alert data input? 

within 10 to 30 

o 

o 

o 

o 

o 

o 

o 

minutes of your 
alert data input? 

o 

o 

o 

o 

o 

o 

o 


Q16: Are you likely to use WEA to issue a public alert for a severe and urgent event if prior alerts 
sent by WEA have been disseminated ... _ 



Definitely 

Not 

Very 

Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

to a geographic area 
other than the one 
specified? 

o 

O 

O 

o 

o 

o 

o 

to the specified geo¬ 
graphic area, and also 
to some adjacent 
geographic areas? 

o 

o 

o 

o 

o 

o 

o 

to only a portion of the 
specified geographic 
area? 

o 

o 

o 

o 

o 

o 

o 
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Alert Originator Survey 2 

Q1: A Survey for the Wireless Emergency Alert Service 

Dear Public Safety Colleague: 

In support of the First Responders Group of the U. S. Department of Homeland Security, the 
Software Engineering Institute (SEI) of Carnegie Mellon University is performing research to aid 
Emergency Management Agencies (EMAs) in the adoption and use of the Wireless Emergency 
Alert (WEA) Service (formerly known as the Commercial Mobile Alert Service, or CMAS). If 
you are unfamiliar with WEA, please see the notes below. For WEA to be effective, the public 
must “trust” the alerts that they receive. Furthermore, if alert originators are to use the WEA ser¬ 
vice, they must “trust” it perform properly. To explore these issues of trast, the SEI is conducting 
surveys of the public, and surveys of the alert origination community. 

As a member of the public safety community, we ask you participate in this voluntary survey to 
assist us in our efforts to enhance public safety. Completion of the survey should require less than 
10 minutes. Participation is limited to individuals 18 years of age or older. As a reward for your 
participation, survey respondents will receive instructions to access a free, follow-up, invitation- 
only webinar sharing further details of the WEA system. 

Thank you for your assistance in this important study. 

Notes on the WEA Service 

WEA is a partnership between the FCC, FEMA and wireless carriers. WEA enables federal, state, 
and local government authorities to send geographically targeted, text-like alerts to the public via 
wireless mobile devices such as cell phones and tablets. This service can be used to deliver alerts 
from the President of the United States, AMBER alerts, and alerts regarding imminent local 
threats such as floods, chemical spills, civil unrest, etc. These alerts are sent to all compatible mo¬ 
bile devices within the geographic area specified by the alert originator. For more information 
regarding WEA, please see http://www.fema.gov/integrated-public-alert-waming-system. 

Q2: Online Consent 

This survey is part of a research study conducted by Robert W. Stoddard II at Carnegie Mellon 
University. The purpose of the research is to study the factors that may influence the public’s or 
alert originator’s tmst in the soon-to-be-deployed United States Wireless Emergency Alert (WEA) 
notification system. 

Procedures 

Survey recipients will be asked approximately 12 questions regarding their reactions to different 
aspects and content of potential alert messages. The survey is expected to take no more than 10 
minutes. 

Participant Requirements 

Participation in this study is limited to individuals age 18 and older. Participants must currently 
use some form of a cellular phone. 
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Risks 


The discomfort and risk associated with participation in this study are no greater than those ordi¬ 
narily encountered in daily life or during other online activities. Discomfort may arise from the 
need to maintain focus and attention when responding to the 20 multiple choice questions that 
include repetition and subtle differences within the message content. The risk of a breach of con¬ 
fidentiality resulting in the inadvertent release of email addresses will be managed by the re¬ 
searcher by ensuring secure access to this information through password control of the online 
survey tool, subsequent removal of the information from the online system, and archival storage 
in a secured filing cabinet under control of the researcher. 

Benefits 

There may be no personal benefit from your participation in the study but the knowledge received 
may be of value to humanity. Respondents will be notified of a future, free, invitation-only webi¬ 
nar providing more detail of the Wireless Emergency Alert system. 

Compensation & Costs 

There is no compensation for participation in this study. There will be no cost to you if you partic¬ 
ipate in this study. 

Confidentiality 

By participating in this research, you understand and agree that Carnegie Mellon may be required 
to disclose your consent form, data and other personally identifiable information as required by 
law, regulation, subpoena or court order. Otherwise, your confidentiality will be maintained in the 
following manner: Your data and consent form will be kept separate. Your consent form will be 
stored in a locked location on Carnegie Mellon property and will not be disclosed to third parties. 
By participating, you understand and agree that the data and information gathered during this 
study may be used by Carnegie Mellon and published and/or disclosed by Carnegie Mellon to 
others outside of Carnegie Mellon. However, your name, address, contact information and other 
direct personal identifiers in your consent form will not be mentioned in any such publication or 
dissemination of the research data and/or results by Carnegie Mellon. 

Right to Ask Questions & Contact Information 

If you have any questions about this study, you should feel free to ask them by contacting the 
Principal Investigator now at Robert W. Stoddard II, Principal Researcher, Software Engineering 
Institute, Carnegie Mellon University, 4500 Fifth Avenue (Office 3110), Pittsburgh, PA 15213, 
412-268-1121, rws@sei.cmu.edu]. If you have questions later, desire additional information, or 
wish to withdraw your participation please contact the Principle Investigator by mail, phone or e- 
mail in accordance with the contact information listed above. 

If you have questions pertaining to your rights as a research participant; or to report objections to 
this study, you should contact the Research Regulatory Compliance Office at Carnegie Mellon 
University. Email: irb-review@andrew.cmu.edu . Phone: 412-268-1901 or 412-268-5460. 
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Voluntary Participation 

Your participation in this research is voluntary. You may discontinue participation at any time 
during the research activity. 

Q3: Please answer the following three questions related to your consent. If you answer “No” to 
any question, you will be ineligible to take the survey. _ 



Yes 

No 

Are you age 18 or older? 

o 

o 

Have you read and do you understand the previous online consent information? 

o 

o 

Do you want to participate in this research and continue with the survey? 

o 

o 


NOTE: Q4 through Q16 are presented only if the respondent answers “Yes” to all questions in Q3. 


Q4: Instructions 

Each of the following survey questions presents you with a situation, and asks you to estimate the 
likelihood of a particular response to that situation. The choices for your answer are: 

Definitely Not I would respond to the situation less than 5% of the time. 

Very Probably Not I would respond to the situation 5-20% of the time. 

Probably Not I would respond to the situation 20^0% of the time. 

Undecided I would respond to the situation 40-60% of the time. 

Probably I would respond to the situation 60-80% of the time. 

Very Probably I would respond to the situation 80-95% of the time. 

Definitely I would respond to the situation more than 95% of the time. 

Q5: What is the approximate population within your jurisdiction (in thousands)? 

Residents _ 

Transients _ 


Q6: Does your agency presently issue public alerts via email, Emergency Notification System, 
EAS, or other mechanism? 

o No 
o Yes 

o Don’t Know 
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Q7: Do you think WEA would be an appropriate tool to issue a public alert for an event that 
poses ■■■ _ 



Definitely 

Not 

Very Prob¬ 
ably Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

an extraordinary 
threat to life? 

o 

o 

o 

o 

o 

o 

o 

significant threat to 
life? 

o 

o 

o 

o 

o 

o 

o 

a possible threat to 
life? 

o 

o 

o 

o 

o 

o 

o 

an extraordinary 
threat to property? 

o 

o 

o 

o 

o 

o 

o 

a significant threat 
to property? 

o 

o 

o 

o 

o 

o 

o 

a possible threat to 
property? 

o 

o 

o 

o 

o 

o 

o 


Q8: Do you think WEA would be an appropriate tool to issue a public alert for a severe and 
urgent event _ 



Definitely 

Not 

Very Proba¬ 
bly Not 

Probably 

Not 

Maybe 

Probably 

Very Prob¬ 
ably 

Definitely 

that occurs at 
10:30 AM 

o 

o 

o 

o 

o 

o 

o 

that occurs at 
6:30 PM 

o 

o 

o 

o 

o 

o 

o 

that occurs at 
2:30 AM 

o 

o 

o 

o 

o 

o 

o 


Q9: Are you likely to use WEA to issue public alerts if you can access it 



Definitely 

Not 

Very 

Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

from several designated 
facilities (e.g., your 
Emergency Manage¬ 
ment office, police pre¬ 
cinct stations, etc.) 
within your jurisdiction, 
as well as remotely from 
your mobile device(s)? 

o 

o 

o 

o 

o 

o 

o 

only from several desig¬ 
nated facilities (e.g., 
your Emergency Man¬ 
agement office, police 
precinct stations, etc.) 
within your jurisdiction? 

o 

o 

o 

o 

o 

o 

o 

only from your primary 
office? 

o 

o 

o 

o 

o 

o 

o 
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Q10: Are you likely to use WEA to issue a public alert for a severe and urgent event if the 
process of creating and issuing an alert _ 



Definitely 

Not 

Very Proba¬ 
bly Not 

Probably 

Not 

Maybe 

Probably 

Very Prob¬ 
ably 

Definitely 

takes 5 
minutes 

o 

o 

o 

o 

o 

o 

o 

takes 10 
minutes 

o 

o 

o 

o 

o 

o 

o 

takes 20 
minutes 

o 

o 

o 

o 

o 

o 

o 

takes 40 
minutes 

o 

o 

o 

o 

o 

o 

o 


Q11: Are you likely to use WEA to issue a public alert fora severe and urgent event 



Definitely 

Not 

Very 

Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

if your message crea¬ 
tion process inciudes 
the use of tempiates to 
assist you in rapidiy 
creating an accurate 
aiert? 

o 

o 

o 

o 

o 

o 

o 

if your message crea¬ 
tion process does not 
inciude the use of tem¬ 
piates to assist you in 
rapidiy creating an 
accurate aiert? 

o 

o 

o 

o 

o 

o 

o 


Q12: Are you likely to use WEA to issue a public alert if opportunities to practice the skills 
needed to use the WEA service are ... 



Definitely 

Not 

Very Prob¬ 
ably Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

Frequent (e.g., 
twice per week) ? 

o 

o 

o 

o 

o 

o 

o 

Occasionai (e.g., 
twice per month) ? 

o 

o 

o 

o 

o 

o 

o 

Rare (e.g., twice 
per year) ? 

o 

o 

o 

o 

o 

o 

o 
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Q13: Are you likely to use WEA to issue a public alert if 



Definitely 

Not 

Very 

Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

you have been unable 
to verify timely and 
accurate transmission 
of prior alerts by the 
WEA service? 

O 

O 

o 

o 

o 

o 

o 

you have verified time¬ 
ly and accurate trans¬ 
mission of prior alerts 
by the WEA service? 

o 

o 

o 

o 

o 

o 

o 

you have verified that 
prior alerts have not 
been transmitted in a 
timely and accurate 
manner? 

o 

o 

o 

o 

o 

o 

o 


Q14: Are you likely to use WEA to issue a public alert for a severe and urgent event if prior alerts 


have been disseminated ... 



Definitely 

Not 

Very Prob¬ 
ably Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

within 2 minutes of 








your alert data 
input? 

within 2 to 5 

o 

o 

o 

o 

o 

o 

o 

minutes of your 
alert data input? 

within 5 to 10 

o 

o 

o 

o 

o 

o 

o 

minutes of your 
alert data input? 

within 10 to 30 

o 

o 

o 

o 

o 

o 

o 

minutes of your 
alert data input? 

o 

o 

o 

o 

o 

o 

o 
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Q15: Are you likely to use WEA to issue a public alert fora severe and urgent event if 



Definitely 

Very 

Probably 

Maybe 

Probably 

Very 

Definitely 


Not 

Probably 

Not 

Not 



Probably 


you are able to issue 
only "standardized” 
messages developed 
by the WEA service 
based on your CAP 
inputs? 

o 

o 

o 

o 

o 

o 

o 

you can craft any mes¬ 
sage of your choosing 
with a maximum size of 
90 characters? 

o 

o 

o 

o 

o 

o 

o 

you can craft any mes¬ 
sage of your choosing 
with a maximum size of 

180 characters? 

o 

o 

o 

o 

o 

o 

o 

you can craft any mes¬ 
sage of your choosing 
with a maximum size of 
270 characters? 

o 

o 

o 

o 

o 

o 

o 


Q16: Are you likely to use WEA to issue a public alert for a severe and urgent event if the 
public... _ 



Definitely 

Very 

Probably 

Maybe 

Probably 

Very 

Definitely 


Not 

Probably 

Not 

Not 



Probably 


has been previously 
informed about WEA 
and its capabilities via 
public media such as 
newspaper reports, TV 
news, radio news, 
and/or your own social 
media channels (e.g., 
web site, Facebook 
page, Twitter account)? 

o 

o 

o 

o 

o 

o 

o 

has not been previously 
informed about WEA 
and its capabilities via 
public media such as 
newspaper reports, TV 
news, radio news, 
and/or your own social 
media channels (e.g., 
web site, Facebook 
page, Twitter account)? 

o 

o 

o 

o 

o 

o 

o 
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Alert Originator Survey 3 

Q1: A Survey for the Wireless Emergency Alert Service 

Dear Public Safety Colleague: 

In support of the First Responders Group of the U.S. Department of Homeland Security, the 
Software Engineering Institute (SEI) of Carnegie Mellon University is performing research to aid 
Emergency Management Agencies (EMAs) in the adoption and use of the Wireless Emergency 
Alert (WEA) Service (formerly known as the Commercial Mobile Alert Service, or CMAS). If 
you are unfamiliar with WEA, please see the notes below. For WEA to be effective, the public 
must “trust” the alerts that they receive. Furthermore, if alert originators are to use the WEA ser¬ 
vice, they must “tmsf ’ it perform properly. To explore these issues of trast, the SEI is conducting 
surveys of the public, and surveys of the alert origination community. 

As a member of the public safety community, we ask you participate in this voluntary survey to 
assist us in our efforts to enhance public safety. Completion of the survey should require less than 
10 minutes. Participation is limited to individuals 18 years of age or older. As a reward for your 
participation, survey respondents will receive instructions to access a free, follow-up, invitation- 
only webinar sharing further details of the WEA system. 

Thank you for your assistance in this important study. 

Notes on the WEA Service 

WEA is a partnership between the FCC, FEMA and wireless carriers. WEA enables federal, state, 
and local government authorities to send geographically targeted, text-like alerts to the public via 
wireless mobile devices such as cell phones and tablets. This service can be used to deliver alerts 
from the President of the United States, AMBER alerts, and alerts regarding imminent local 
threats such as floods, chemical spills, civil unrest, etc. These alerts are sent to all compatible mo¬ 
bile devices within the geographic area specified by the alert originator. For more information 
regarding WEA, please see http://www.fema.gov/integrated-public-alert-waming-system. 

Q2: Online Consent 

This survey is part of a research study conducted by Robert W. Stoddard II at Carnegie Mellon 
University. The purpose of the research is to study the factors that may influence the public’s or 
alert originator’s tmst in the soon-to-be-deployed United States Wireless Emergency Alert (WEA) 
notification system. 

Procedures 

Survey recipients will be asked approximately 12 questions regarding their reactions to different 
aspects and content of potential alert messages. The survey is expected to take no more than 10 
minutes. 

Participant Requirements 

Participation in this study is limited to individuals age 18 and older. Participants must currently 
use some form of a cellular phone. 


CMU/SEI-2013-SR-021 | 144 


Risks 


The discomfort and risk associated with participation in this study are no greater than those ordi¬ 
narily encountered in daily life or during other online activities. Discomfort may arise from the 
need to maintain focus and attention when responding to the 20 multiple choice questions that 
include repetition and subtle differences within the message content. The risk of a breach of con¬ 
fidentiality resulting in the inadvertent release of email addresses will be managed by the re¬ 
searcher by ensuring secure access to this information through password control of the online 
survey tool, subsequent removal of the information from the online system, and archival storage 
in a secured filing cabinet under control of the researcher. 

Benefits 

There may be no personal benefit from your participation in the study but the knowledge received 
may be of value to humanity. Respondents will be notified of a future, free, invitation-only webi¬ 
nar providing more detail of the Wireless Emergency Alert system. 

Compensation & Costs 

There is no compensation for participation in this study. There will be no cost to you if you partic¬ 
ipate in this study. 

Confidentiality 

By participating in this research, you understand and agree that Carnegie Mellon may be required 
to disclose your consent form, data and other personally identifiable information as required by 
law, regulation, subpoena or court order. Otherwise, your confidentiality will be maintained in the 
following manner: Your data and consent form will be kept separate. Your consent form will be 
stored in a locked location on Carnegie Mellon property and will not be disclosed to third parties. 
By participating, you understand and agree that the data and information gathered during this 
study may be used by Carnegie Mellon and published and/or disclosed by Carnegie Mellon to 
others outside of Carnegie Mellon. However, your name, address, contact information and other 
direct personal identifiers in your consent form will not be mentioned in any such publication or 
dissemination of the research data and/or results by Carnegie Mellon. 

Right to Ask Questions & Contact Information 

If you have any questions about this study, you should feel free to ask them by contacting the 
Principal Investigator now at Robert W. Stoddard II, Principal Researcher, Software Engineering 
Institute, Carnegie Mellon University, 4500 Fifth Avenue (Office 3110), Pittsburgh, PA 15213, 
412-268-1121, rws@sei.cmu.edu]. If you have questions later, desire additional information, or 
wish to withdraw your participation please contact the Principle Investigator by mail, phone or e- 
mail in accordance with the contact information listed above. 

If you have questions pertaining to your rights as a research participant; or to report objections to 
this study, you should contact the Research Regulatory Compliance Office at Carnegie Mellon 
University. Email: irb-review@andrew.cmu.edu . Phone: 412-268-1901 or 412-268-5460. 
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Voluntary Participation 

Your participation in this research is voluntary. You may discontinue participation at any time 
during the research activity. 

Q3: Please answer the following three questions related to your consent. If you answer “No” to 
any question, you will be ineligible to take the survey. _ 



Yes 

No 

Are you age 18 or older? 

o 

o 

Have you read and do you understand the previous online consent information? 

o 

o 

Do you want to participate in this research and continue with the survey? 

o 

o 


NOTE: Q4 through Q16 are presented only if the respondent answers “Yes” to all questions in Q3. 


Q4: Instructions 

Each of the following survey questions presents you with a situation, and asks you to estimate the 
likelihood of a particular response to that situation. The choices for your answer are: 

Definitely Not I would respond to the situation less than 5% of the time. 

Very Probably Not I would respond to the situation 5-20% of the time. 

Probably Not I would respond to the situation 20^0% of the time. 

Undecided I would respond to the situation 40-60% of the time. 

Probably I would respond to the situation 60-80% of the time. 

Very Probably I would respond to the situation 80-95% of the time. 

Definitely I would respond to the situation more than 95% of the time. 

Q5: What is the approximate population within your jurisdiction (in thousands)? 

Residents _ 

Transients _ 


Q6: Does your agency presently issue public alerts via email, Emergency Notification System, 
EAS, or other mechanism? 

o No 
o Yes 

o Don’t know 
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Q7: Do you think WEA would be an appropriate tool to issue a public alert for a severe and 
urgent event that is _ 



Definitely 

Not 

Very Proba¬ 
bly Not 

Probably 

Not 

Maybe 

Probably 

Very Prob¬ 
ably 

Definitely 

30% likely to 
occur? 

o 

o 

o 

o 

o 

o 

o 

50% likely to 
occur? 

o 

o 

o 

o 

o 

o 

o 

70% likely to 
occur? 

o 

o 

o 

o 

o 

o 

o 

90% likely to 
occur? 

o 

o 

o 

o 

o 

o 

o 


Q8: Public alerting responsibility is often distributed across multiple emergency management 
agencies (EMAs) based on the type of alert and location. However, this distribution is not always 
foolproof due to unclear alert classifications and overlapping jurisdictions (e.g., the jurisdiction 
of a municipal EMA is within the jurisdiction of a county EMA within the jurisdiction of a state 
EMA). In some cases, it may be unclear whether or not your EMA has the primary responsibility 
for issuing an alert. How likely are you to use WEA for an urgent and severe event in your 
jurisdiction ... _ 



Definitely 

Not 

Very 

Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

if you are 50% sure 
that the responsibility 
for issuing the alert 
rests with your EMA 
and not another? 

o 

o 

o 

o 

o 

o 

o 

if you are 70% sure 
that the responsibility 
for issuing the alert 
rests with your EMA 
and not another? 

o 

o 

o 

o 

o 

o 

o 

if you are 90% sure 
that the responsibility 
for issuing the alert 
rests with your EMA 
and not another? 

o 

o 

o 

o 

o 

o 

o 

if you are 99% sure 
that the responsibility 
for issuing the alert 
rests with your EMA 
and not another? 

o 

o 

o 

o 

o 

o 

o 
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Q9: Are you likely to use WEA to issue a public alert for a severe and urgent event if the service 



Definitely 

Not 

Very 

Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

successfully trans¬ 
mits 99.9% of the 
alerts submitted 

o 

o 

o 

o 

o 

o 

o 

successfully trans¬ 
mits 99% of the 
alerts submitted 

o 

o 

o 

o 

o 

o 

o 

successfully trans¬ 
mits 90% of the 
alerts submitted 

o 

o 

o 

o 

o 

o 

o 


Q10: Are you likely to use WEA to issue a public alert for a severe and urgent event if the 
process of creating and issuing an alert _ 



Definitely 

Not 

Very Proba¬ 
bly Not 

Probably 

Not 

Maybe 

Probably 

Very Prob¬ 
ably 

Definitely 

takes 5 
minutes 

o 

o 

o 

o 

o 

o 

o 

takes 10 
minutes 

o 

o 

o 

o 

o 

o 

o 

takes 20 
minutes 

o 

o 

o 

o 

o 

o 

o 

takes 40 
minutes 

o 

o 

o 

o 

o 

o 

o 


Q11: Are you likely to use WEA to issue a public alert if you have received 


Definitely 

Not 

Very 

Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

extensive training 
(e.g., 40 hours) on 
the use of the WEA 

o 

o 

o 

o 

o 

o 

o 

service 








adequate training 
(e.g., 16 hours) on 
the use of the WEA 

o 

o 

o 

o 

o 

o 

o 

service 








minimal training (4 
hours) on the use of 
the WEA service 

o 

o 

o 

o 

o 

o 

o 


CMU/SEI-2013-SR-021 | 148 



































Q12: Are you likely to use the WEA service if you are aware of 



Definitely 

Not 

Very 

Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

no cyber attacks (e.g., 
spoofing, tampering, 
denial of service) on the 
WEA service reported 
by any WEA users with¬ 
in the past 12 months? 

o 

o 

o 

o 

o 

o 

o 

unsuccessful cyber 
attacks (e.g., failed 
attempts at spoofing, 
tampering, denial of 
service) on the WEA 
service reported by any 
WEA users within the 
past 12 months? 

o 

o 

o 

o 

o 

o 

o 

successful cyber attacks 
(e.g., spoofing, tamper¬ 
ing, denial of service) on 
the WEA service report¬ 
ed by any WEA users 
within the past 12 
months? 

o 

o 

o 

o 

o 

o 

o 

successful cyber attacks 
(e.g., spoofing, tamper¬ 
ing, denial of service) on 
the WEA service report¬ 
ed by your agency with¬ 
in the past 12 months? 

o 

o 

o 

o 

o 

o 

o 


Q13: Are you likely to use WEA to issue a public alert for a severe and urgent event if public 
feedback resulting from prior alerts has been _ 



Definitely 

Not 

Very Proba¬ 
bly Not 

Probably 

Not 

Maybe 

Probably 

Very Prob¬ 
ably 

Definitely 

unfavorable? 

o 

o 

O 

o 

o 

o 

o 

neutral? 

o 

o 

o 

o 

o 

o 

o 

favorable? 

o 

o 

o 

o 

o 

o 

o 


CMU/SEI-2013-SR-021 | 149 
























Q14: Are you likely to use WEA to issue a public alert for a severe and urgent event if prior alerts 


have been disseminated ... 



Definitely 

Not 

Very Prob¬ 
ably Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

within 2 minutes of 








your alert data 
input? 

within 2 to 5 

o 

o 

O 

o 

o 

o 

o 

minutes of your 
alert data input? 

within 5 to 10 

o 

o 

o 

o 

o 

o 

o 

minutes of your 
alert data input? 

within 10 to 30 

o 

o 

o 

o 

o 

o 

o 

minutes of your 
alert data input? 

o 

o 

o 

o 

o 

o 

o 


Q15: Are you likely to use WEA to issue a public alert for a severe and urgent event if prior alerts 
sent by WEA have been disseminated ... _ 



Definitely 

Not 

Very 

Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

with no errors in the 
message data? 

O 

o 

o 

o 

o 

o 

o 

with minor errors in the 
message data (e.g., 
errors that do not affect 
the understandability or 
content of the mes¬ 
sage)? 

o 

o 

o 

o 

o 

o 

o 

with significant errors in 
the message data (e.g., 
errors that affect the 
understandability or 
content of the mes¬ 
sage)? 

o 

o 

o 

o 

o 

o 

o 


Q16: Are you likely to use WEA to issue a public alert for a severe and urgent event if you have 
already used WEA to issue 2 alerts ... _ 



Definitely 

Not 

Very Proba¬ 
bly Not 

Probably 

Not 

Maybe 

Probably 

Very Prob¬ 
ably 

Definitely 

within the past 
week? 

within the past 
month? 

within the past 
year? 

0 0 0 

0 0 0 

0 0 0 

0 0 0 

0 0 0 

0 0 0 

0 0 0 
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Appendix M Mapping of Aiert Originator Trust Modei Reiationships to Survey Questions 


AS 

Driver => Responder 

Method 

Qualtrics ID 

Q# 

Question 

A1 

Urgency => Appropriate 

delta 

S1Q7_1 

1 

Do you think WEA would be an appropriate tool to issue a public alert for an event that requires public 
action within 10 minutes? 

S1Q7_2 

2 

Do you think WEA would be an appropriate tool to issue a public alert for an event that requires public 
action within 30 minutes? 

S1Q7_3 

3 

Do you think WEA would be an appropriate tool to issue a public alert for an event that requires public 
action within 60 minutes? 

S1Q7_4 

4 

Do you think WEA would be an appropriate tool to issue a public alert for an event that requires public 
action within 2 hours? 

A2 

Severity => Appropriate 

delta 

S1Q7_1 

5 

Do you think WEA would be an appropriate tool to issue a public alert for an event that poses extraordi¬ 
nary threat to life? 

S1Q7_2 

6 

Do you think WEA would be an appropriate tool to issue a public alert for an event that poses significant 
threat to life? 

S1Q7_3 

7 

Do you think WEA would be an appropriate tool to issue a public alert for an event that poses possible 
threat to life? 

S1Q7_4 

8 

Do you think WEA would be an appropriate tool to issue a public alert for an event that poses extraordi¬ 
nary threat to property? 

S1Q7_5 

9 

Do you think WEA would be an appropriate tool to issue a public alert for an event that poses significant 
threat to property? 

S1Q7_6 

10 

Do you think WEA would be an appropriate tool to issue a public alert for an event that poses possible 
threat to property? 

A3 

Certainty => Appropriate 

delta 

S3Q7_1 

11 

Do you think WEA would be an appropriate tool to issue a public alert for a severe and urgent event that 
is 30% likely to occur? 

S3Q7_2 

12 

Do you think WEA would be an appropriate tool to issue a public alert for a severe and urgent event that 
is 50% likely to occur? 

S3Q7_3 

13 

Do you think WEA would be an appropriate tool to issue a public alert for a severe and urgent event that 
is 70% likely to occur? 

S3Q7_4 

14 

Do you think WEA would be an appropriate tool to issue a public alert for a severe and urgent event that 
is 90% likely to occur? 
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Mt 

Driver => Responder 

Method 

Qualtrics ID 

Q# 

Question 

A4 

Geographic . . , 

breadth ^ Appropriate 

delta 

S1Q8_1 

15 

Do you think WEA would be an appropriate tool to issue a public alert for a severe and urgent event 
where 30% of the alert recipients are in the hazard zone and 70% are outside the zone? 

S1Q8_2 

16 

Do you think WEA would be an appropriate tool to issue a public alert for a severe and urgent event 
where 50% of the alert recipients are in the hazard zone and 50% are outside the zone? 

S1Q8_3 

17 

Do you think WEA would be an appropriate tool to issue a public alert for a severe and urgent event 
where 70% of the alert recipients are in the hazard zone and 30% are outside the zone? 

S1Q8_4 

18 

Do you think WEA would be an appropriate tool to issue a public alert for a severe and urgent event 
where 90% of the alert recipients are in the hazard zone and 10% are outside the zone? 

A5 

Time of day => Appropriate 

delta 

S1Q8_1 

19 

Do you think WEA would be an appropriate tool to issue a public alert for a severe and urgent event that 
occurs at 10:30 AM? 

S1Q8_2 

20 

Do you think WEA would be an appropriate tool to issue a public alert for a severe and urgent event that 
occurs at 6:30 PM? 

S1Q8_3 

21 

Do you think WEA would be an appropriate tool to issue a public alert for a severe and urgent event that 
occurs at 2:30 AM? 
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Mt 

Driver 

=> Responder 

Me¬ 

thod 

Qual- 
trics ID 

Q# 

Question 





S3Q8_1 

22 

Public alerting responsibility is often distributed across multiple emergency management agencies (EMAs) 
based on the type of alert and location. However, this distribution is not always foolproof due to unclear alert 
classifications and overlapping jurisdictions (e.g., the jurisdiction of a municipal EMA is within the jurisdiction of 
a county EMA within the jurisdiction of a state EMA). In some cases, it may be unclear whether or not your 

EMA has the primary responsibility for issuing an alert. How likely are you to use WEA for an urgent and se¬ 
vere event in your jurisdiction if you are 50% sure that the responsibility for issuing the alert rests with your 

EMA and not another? 

A6 

Responsibility 

=> Appropriate 

delta 

S3Q8_2 

23 

Public alerting responsibility is often distributed across multiple emergency management agencies (EMAs) 
based on the type of alert and location. However, this distribution is not always foolproof due to unclear alert 
classifications and overlapping jurisdictions (e.g., the jurisdiction of a municipal EMA is within the jurisdiction of 
a county EMA within the jurisdiction of a state EMA). In some cases, it may be unclear whether or not your 

EMA has the primary responsibility for issuing an alert. How likely are you to use WEA for an urgent and se¬ 
vere event in your jurisdiction if you are 70% sure that the responsibility for issuing the alert rests with your 

EMA and not another? 

S3Q8_3 

24 

Public alerting responsibility is often distributed across multiple emergency management agencies (EMAs) 
based on the type of alert and location. However, this distribution is not always foolproof due to unclear alert 
classifications and overlapping jurisdictions (e.g., the jurisdiction of a municipal EMA is within the jurisdiction of 
a county EMA within the jurisdiction of a state EMA). In some cases, it may be unclear whether or not your 

EMA has the primary responsibility for issuing an alert. How likely are you to use WEA for an urgent and se¬ 
vere event in your jurisdiction if you are 90% sure that the responsibility for issuing the alert rests with your 

EMA and not another? 





S3Q8_4 

25 

Public alerting responsibility is often distributed across multiple emergency management agencies (EMAs) 
based on the type of alert and location. However, this distribution is not always foolproof due to unclear alert 
classifications and overlapping jurisdictions (e.g., the jurisdiction of a municipal EMA is within the jurisdiction of 
a county EMA within the jurisdiction of a state EMA). In some cases, it may be unclear whether or not your 

EMA has the primary responsibility for issuing an alert. How likely are you to use WEA for an urgent and se¬ 
vere event in your jurisdiction if you are 99% sure that the responsibility for issuing the alert rests with your 

EMA and not another? 





S1Q9_1 

26 

Are you likely to use WEA to issue a public alert for a severe and urgent event if the WEA service is typically 
unavailable for use 1 hour per week (i.e., 99.4% availability) for maintenance? 

A7 

System 

readiness 

=> Availability 

delta 

S1Q9_2 

27 

Are you likely to use WEA to issue a public alert for a severe and urgent event if the WEA service is typically 
unavailable for use 1 hour per month (i.e., 99.9% availability) for maintenance? 





S1Q9_3 

28 

Are you likely to use WEA to issue a public alert for a severe and urgent event if the WEA service is typically 
unavailable for use 1 hour per year (i.e., 99.99% availability) for maintenance? 
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A# 

Driver 

=> 

Responder 

Me¬ 

thod 

Qual- 
trics ID 

Q# 

Question 

A8 

System 


Availability 

delta 

S2Q9_1 


Are you likely to use WEA to issue public alerts if you can access it from several designated facilities (e.g., 
your Emergency Management office, police precinct stations, etc.) within your jurisdiction, as well as remotely 
from your mobile device(s)? 

accessibility 


S2Q9_2 

30 

Are you likely to use WEA to issue public alerts if you can access it only from several designated facilities 
(e.g., your Emergency Management office, police precinct stations, etc.) within your jurisdiction? 






S2Q9 3 

31 

Are you likely to use WEA to issue public alerts if you can access it only from your primary office? 






S3Q9_1 

32 

Are you likely to use WEA to issue a public alert for a severe and urgent event if the service successfully 
transmits 99.9% of the alerts submitted? 

A9 

System reli¬ 
ability 


Availability 

delta 

S3Q9_2 

33 

Are you likely to use WEA to issue a public alert for a severe and urgent event if the service successfully 
transmits 99% of the alerts submitted? 






S3Q9_3 

34 

Are you likely to use WEA to issue a public alert for a severe and urgent event if the service successfully 
transmits 90% of the alerts submitted? 






SI,2,3 
Q10 1 

35 

Are you likely to use WEA to issue a public alert for a severe and urgent event if the process of creating and 
issuing an alert takes 5 minutes? 

A10 

Magnitude of 


Availability 

delta 

SI,2,3 
Q10 2 

36 

Are you likely to use WEA to issue a public alert for a severe and urgent event if the process of creating and 
issuing an alert takes 10 minutes? 

effort 


SI,2,3 
Q10 3 

37 

Are you likely to use WEA to issue a public alert for a severe and urgent event if the process of creating and 
issuing an alert takes 20 minutes? 






SI,2,3 
Q10 4 

38 

Are you likely to use WEA to issue a public alert for a severe and urgent event if the process of creating and 
issuing an alert takes 40 minutes? 

All 

Cross¬ 

system 

integration 


Availability 

delta 

S1Q11_1 

39 

Are you likely to use WEA to issue a public alert for a severe and urgent event if the creation and issuance of 
a WEA alert requires using a system that is independent of other alerting and/or emergency management 
systems in your office? 


S1Q11_2 

40 

Are you likely to use WEA to issue a public alert for a severe and urgent event if the creation and issuance of 
a WEA alert is accomplished within a system that is integrated with other alerting and/or emergency man¬ 
agement systems in your office? 

A12 

Templates 


Availability 

delta 

S2Q11_1 

41 

Are you likely to use WEA to issue a public alert for a severe and urgent event if your message creation pro¬ 
cess includes the use of templates to assist you in rapidly creating an accurate alert? 


S2Q11_2 

42 

Are you likely to use WEA to issue a public alert for a severe and urgent event if your message creation pro¬ 
cess does not include the use of templates to assist you in rapidly creating an accurate alert? 
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A# 

Driver => 

Responder 

Me¬ 

thod 

Qual- 
trics ID 

Q# 

Question 


Skills/ 

competen- => 

cies 



S3Q11_1 

43 

Are you likely to use WEA to issue a public alert if you have received extensive training (e.g. 40 hours) on the 
use of the WEA service? 

A13 

Availability 

delta 

S3Q11_2 

44 

Are you likely to use WEA to issue a public alert if you have received adequate training (e.g., 16 hours) on the 
use of the WEA service? 




S3Q11_3 

45 

Are you likely to use WEA to issue a public alert if you have received minimal training (e.g., 4 hours) on the 
use of the WEA service? 





S1Q12_1 

46 

Are you likely to use WEA to issue a public alert If you believe you have a thorough understanding of the 
principles and applications of the WEA service? 

A14 

Understand¬ 

ing 

Availability 

delta 

S1Q12_2 

47 

Are you likely to use WEA to issue a public alert if you believe you have a moderate understanding of the 
principles and applications of the WEA service? 





S1Q12_3 

48 

Are you likely to use WEA to issue a public alert if you believe you have minimal understanding of the princi¬ 
ples and applications of the WEA service? 





S2Q12_1 

49 

Are you likely to use WEA to issue a public alert If opportunities to practice the skills needed to use the WEA 
service are frequent (e.g., twice per week)? 

A15 

Practice => 

Availability 

delta 

S2Q12_2 

50 

Are you likely to use WEA to issue a public alert if opportunities to practice the skills needed to use the WEA 
service are occasional (e.g., twice per month)? 





S2Q12_3 

51 

Are you likely to use WEA to issue a public alert if opportunities to practice the skills needed to use the WEA 
service are rare (e.g., twice per year)? 





S3Q12_1 

52 

Are you likely to use WEA if you are aware of no cyber attacks (e.g., spoofing, tampering, denial of service) 
on the WEA service reported by any WEA users within the past 12 months? 

A16 

Security => 

Availability 

delta 

S3Q12_2 

53 

Are you likely to use WEA if you are aware of unsuccessful cyber attacks (e.g., failed attempts at spoofing, 
tampering, denial of service) on the WEA service reported by any WEA users within the past 12 months? 

S3Q12_3 

54 

Are you likely to use WEA if you are aware of successful cyber attacks (e.g., spoofing, tampering, denial of 
service) on the WEA service reported by other WEA users within the past 12 months? 





S3Q12_4 

55 

Are you likely to use WEA if you are aware of successful cyber attacks (e.g., spoofing, tampering, denial of 
service) on the WEA service reported by your agency within the past 12 months? 
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A# 

Driver 

=> 

Responder 

Me¬ 

thod 

Qualtrics 

ID 

Q# 

Question 






S1Q13_1 

56 

Would you be confident that the WEA service had delivered your alert if you received no response from the 
WEA service regarding the status of your alert from IPAWS-OPEN? 






S1Q13_2 

57 

Would you be confident that the WEA service had delivered your alert if you received a response from the 
WEA service indicating that your message had been received by IPAWS-OPEN? 

A17 

Real-time 

system 

feedback 


Effectiveness 

delta 

S1Q13_3 

58 

Would you be confident that the WEA service had delivered your alert if you received a response from the 
WEA service indicating that your message had been received and accepted by IPAWS-OPEN? 


S1Q13_4 

59 

Would you be confident that the WEA service had delivered your alert if you received a response from the 
WEA service indicating that your message had been received and accepted by IPAWS-OPEN, and sent to 
the wireless carriers? 






S1Q13_5 

60 

Would you be confident that the WEA service had delivered your alert if you received a response from the 
WEA service indicating that your message had been received and accepted by IPAWS-OPEN, sent to the 
wireless carriers, and transmitted by the wireless carriers? 


Historical 

system 

feedback 




S2Q13_1 

61 

Are you likely to use WEA to issue a public alert if you have been unable to verify timely and accurate 
transmission of prior alerts by the WEA service? 

A18 


Effectiveness 

delta 

S2Q13_2 

62 

Are you likely to use WEA to issue a public alert if you have verified timely and accurate transmission of prior 
alerts by the WEA service? 





S2Q13_3 

63 

Are you likely to use WEA to issue a public alert if you have verified that prior alerts have not been transmit¬ 
ted in a timely and accurate manner? 


Public 

feedback 

history 




S3Q13_1 

64 

Are you likely to use WEA to issue a public alert for a severe and urgent event if public feedback resulting 
from prior alerts has been unfavorable? 

A19 


Effectiveness 

delta 

S3Q13_2 

65 

Are you likely to use WEA to issue a public alert for a severe and urgent event if public feedback resulting 
from prior alerts has been neutral? 





S3Q13_3 

66 

Are you likely to use WEA to issue a public alert for a severe and urgent event if public feedback resulting 
from prior alerts has been favorable? 






S1Q14_1 

67 

Are you likely to use WEA to issue a public alert for a severe and urgent event If findings of internal After 

Action Reviews of prior alerts have been unfavorable? 

A20 

After-action 
review data 


Effectiveness 

delta 

S1Q14_2 

68 

Are you likely to use WEA to issue a public alert for a severe and urgent event if findings of internal After 

Action Reviews of prior alerts have been neutral? 






S1Q14_3 

69 

Are you likely to use WEA to issue a public alert for a severe and urgent event if findings of internal After 

Action Reviews of prior alerts have been favorable? 
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A# 

Driver => Responder 

Me¬ 

thod 

Qualtrics 

ID 

Q# 

Question 




S1Q15_1, 
S2,3Q14 1 

70 

Are you likely to use WEA to issue a public alert for a severe and urgent event if prior alerts have been 
disseminated within 2 minutes of your alert data input? 

A21 

Timeliness of 

dissemina- => Effectiveness 

tion 

delta 

S1Q15_2, 
S2,3Q14 2 

71 

Are you likely to use WEA to issue a public alert for a severe and urgent event if prior alerts have been 
disseminated within 2 to 5 minutes of your alert data input? 

S1Q15_3, 
S2,3Q14 3 

72 

Are you likely to use WEA to issue a public alert for a severe and urgent event if prior alerts have been 
disseminated within 5 to 10 minutes of your alert data input? 




S1Q15_4, 
S2,3Q14 4 

73 

Are you likely to use WEA to issue a public alert for a severe and urgent event if prior alerts have been 
disseminated within 10 to 30 minutes of your alert data input? 




S2Q15_1 

74 

Are you likely to use WEA to issue a public alert for a severe and urgent event if you are able to issue only 
‘‘standardized’ messages developed by the WEA service based on your CAP inputs? 

A22 

Message 

under- => Effectiveness 

standability 

delta 

S2Q15_2 

75 

Are you likely to use WEA to issue a public alert for a severe and urgent event if you can craft any mes¬ 
sage of your choosing with a maximum size of 90 characters? 

S2Q15_3 

76 

Are you likely to use WEA to issue a public alert for a severe and urgent event if you can craft any mes¬ 
sage of your choosing with a maximum size of 180 characters? 




S2Q15_4 

77 

Are you likely to use WEA to issue a public alert for a severe and urgent event if you can craft any mes¬ 
sage of your choosing with a maximum size of 270 characters? 




S3Q15_1 

78 

Are you likely to use WEA to issue a public alert for a severe and urgent event if prior alerts sent by WEA 
have been disseminated with no errors in the message data? 

A23 

Message 

=> Effectiveness 

accuracy 

delta 

S3Q15_2 

79 

Are you likely to use WEA to issue a public alert for a severe and urgent event if prior alerts sent by WEA 
have been disseminated with minor errors in the message data (e.g., errors that do not affect the under- 
standability of content of the message)? 




S3Q15_3 

80 

Are you likely to use WEA to issue a public alert for a severe and urgent event if prior alerts sent by WEA 
have been disseminated with significant errors in the message data (errors that affect the understandability 
or content of the message)? 




S1Q16_1 

81 

Are you likely to use WEA to issue a public alert for a severe and urgent event if prior alerts sent by WEA 
have been disseminated to a geographic area other than the one specified? 

A24 

Location 

=> Effectiveness 

accuracy 

delta 

S1Q16_2 

82 

Are you likely to use WEA to issue a public alert for a severe and urgent event if prior alerts sent by WEA 
have been disseminated to the specified geographic area, and also to some adjacent geographic areas? 




S1Q16_3 

83 

Are you likely to use WEA to issue a public alert for a severe and urgent event if prior alerts sent by WEA 
have been disseminated to only a portion of the specified geographic area? 
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A# 

Driver => Responder 

Me¬ 

thod 

Qualtrics 

ID 

Q# 

Question 

A25 

Public 

awareness => Effectiveness 

/outreach 

delta 

S2Q16_1 

84 

Are you likely to use WEA to issue a public alert for a severe and urgent event if the public has been previ¬ 
ously informed about WEA and its capabilities via public media such as newspaper reports, TV news, radio 
news, and/or your own social media channels (e.g., web site, Facebook page, Twitter account)? 

S2Q16_2 

85 

Are you likely to use WEA to issue a public alert for a severe and urgent event if the public has not been 
previously informed about WEA and its capabilities via public media such as newspaper reports, TV news, 
radio news, and/or your own social media channels (e.g., web site, Facebook page, Twitter account)? 

A26 

=> Effectiveness 

frequency 

delta 

S3Q16_1 

86 

Are you likely to use WEA to issue a public alert for a severe and urgent event if you have already used 

WEA to issue 2 alerts within the past week? 

S3Q16_2 

87 

Are you likely to use WEA to issue a public alert for a severe and urgent event if you have already used 

WEA to issue 2 alerts within the past month? 

S3Q16_3 

88 

Are you likely to use WEA to issue a public alert for a severe and urgent event if you have already used 

WEA to issue 2 alerts within the past year? 
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Appendix N Descriptive Statisticai Resuits for Pubiic 
Surveys 


Table 6 portrays the results of the original survey ordinal response. The variable is the Qualtrics 
question identifier. The last digit of the variable name distinguishes which of the three surveys the 
question is from. 


Table 6: Results of the Original Survey Ordinal Responses 


Variable 

N 

W* 

Min 

Q1 

Mdn 

Q3 

Max 

Q5 1 

77 

57 

1.000 

4.000 

5.000 

6.000 

7.000 

Q6 1 

78 

0 

1.000 

5.000 

5.500 

6.000 

7.000 

Q7 1 

78 

0 

1.000 

4.000 

5.000 

6.000 

7.000 

Q8 1 

78 

0 

1.000 

3.000 

4.000 

6.000 

7.000 

Q9 1 

78 

0 

1.000 

4.000 

5.500 

7.000 

7.000 

Q10 1 

78 

0 

1.000 

2.000 

3.000 

5.000 

7.000 

Q11 1 

76 

2 

1.000 

5.000 

6.000 

6.750 

7.000 

Q12 1 

76 

2 

1.000 

4.250 

5.000 

6.000 

7.000 

Q13 1 

76 

2 

1.000 

2.000 

3.500 

4.750 

7.000 

Q14 1 

76 

2 

1.000 

4.000 

5.000 

6.000 

7.000 

Q15 1 

76 

2 

1.000 

4.000 

5.000 

6.000 

7.000 

Q16 1 

76 

2 

1.000 

2.000 

3.000 

4.000 

7.000 

Q17 1 

76 

2 

1.000 

3.250 

5.000 

6.000 

7.000 

Q18 1 

76 

2 

1.000 

3.000 

5.000 

6.000 

7.000 

Q19 1 

75 

3 

1.000 

3.000 

5.000 

6.000 

7.000 

Q20 1 

75 

3 

1.000 

3.000 

5.000 

6.000 

7.000 

Q21 1 

76 

2 

1.000 

3.000 

5.000 

6.000 

7.000 

Q22 1 

76 

2 

1.000 

4.000 

5.000 

6.000 

7.000 

Q23 1 

76 

2 

1.000 

2.000 

3.000 

5.000 

7.000 

Q5 2 

106 

29 

1.000 

1.000 

2.000 

3.000 

6.000 

Q6 2 

105 

1 

1.000 

4.000 

5.000 

6.000 

7.000 

Q7 2 

106 

0 

1.000 

3.000 

5.000 

6.000 

7.000 

Q8 2 

104 

2 

1.000 

3.000 

4.000 

5.000 

7.000 

Q9 2 

104 

2 

1.000 

4.000 

5.000 

6.000 

7.000 

Q10 2 

104 

2 

1.000 

5.000 

6.000 

7.000 

7.000 

Q11 2 

103 

3 

1.000 

5.000 

5.000 

6.000 

7.000 

Q12 2 

104 

2 

1.000 

4.000 

5.000 

6.000 

7.000 

Q13 2 

101 

5 

1.000 

2.000 

3.000 

4.000 

7.000 

Q14 2 

102 

4 

1.000 

4.000 

5.000 

6.000 

7.000 

Q15 2 

100 

6 

1.000 

5.000 

6.000 

6.000 

7.000 

Q16 2 

100 

6 

1.000 

4.000 

5.000 

6.000 

7.000 

Q17 2 

99 

7 

1.000 

4.000 

5.000 

6.000 

7.000 

Q18 2 

98 

8 

1.000 

3.000 

4.000 

6.000 

7.000 

Q19 2 

99 

7 

1.000 

2.000 

3.000 

4.000 

7.000 

Q20 2 

99 

7 

1.000 

5.000 

6.000 

6.000 

7.000 

Q21 2 

98 

8 

1.000 

3.000 

4.000 

5.000 

6.000 

Q22 2 

98 

8 

1.000 

4.000 

5.000 

6.000 

7.000 

Q23 2 

97 

9 

1.000 

4.000 

4.000 

6.000 

7.000 

Q24 2 

97 

9 

1.000 

2.000 

3.000 

3.500 

7.000 

Q5_3 

98 

36 

1.000 

3.000 

5.000 

6.000 

7.000 
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Variable 

N 

N* 

Min 

Q1 

Mdn 

Q3 

Max 

Q6 3 

96 

2 

1.000 

4.000 

5.000 

6.000 

7.000 

Q7 3 

90 

8 

1.000 

4.000 

5.000 

6.000 

7.000 

Q8 3 

96 

2 

1.000 

3.000 

4.000 

5.000 

7.000 

Q9 3 

95 

3 

1.000 

3.000 

4.000 

5.000 

7.000 

Q10 3 

95 

3 

1.000 

5.000 

6.000 

6.000 

7.000 

Q11 3 

95 

3 

1.000 

5.000 

6.000 

6.000 

7.000 

Q12 3 

94 

4 

1.000 

4.000 

5.000 

6.000 

7.000 

Q13 3 

95 

3 

1.000 

3.000 

5.000 

6.000 

7.000 

Q14 3 

95 

3 

1.000 

5.000 

5.000 

6.000 

7.000 

Q15 3 

95 

3 

1.000 

2.000 

4.000 

5.000 

7.000 

Q16 3 

94 

4 

1.000 

2.000 

3.000 

4.000 

6.000 

Q17 3 

94 

4 

1.000 

2.000 

3.000 

4.000 

7.000 

Q18 3 

94 

4 

1.000 

2.000 

3.000 

4.000 

7.000 

Q19 3 

95 

3 

1.000 

3.000 

5.000 

6.000 

7.000 

Q20 3 

95 

3 

1.000 

2.000 

2.000 

4.000 

7.000 

Q21 3 

95 

3 

1.000 

4.000 

5.000 

6.000 

7.000 

Q22 3 

95 

3 

1.000 

3.000 

3.000 

5.000 

7.000 

Q23_3 

95 

3 

1.000 

2.000 

3.000 

3.000 

7.000 
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Appendix O Comparative Anaiysis of Paired Questions for 
Pubiic Surveys 


Table 7: Paired t for CQ10_2 and CQ10_3 



N 

M 

SD 

SEM 

CQ10 2 

67 

78.96 

23.09 

2.82 

CQ10 3 

67 

76.12 

19.99 

2.44 

Difference 

67 

2.84 

30.36 

3.71 


95% upper bound for mean difference: 9.02. 
t Test of mean difference = 0 (vs. < 0): f = 0.76, p = .776. 

Table 8: Paired \.forCQ11_1 and CQ10 2 



N 

M 

SD 

SEM 

CQ11 1 

62 

74.11 

25.17 

3.20 

CQ10 2 

62 

80.08 

21.45 

2.72 

Difference 

62 

-5.97 

32.40 

4.11 


95% upper bound for mean difference: 0.90. 
f Test of mean difference = 0 (vs. < 0): f = -1.45, p = .076. 

Table 9: Paired tfor CQ11_1 and CQ10_3 



N 

M 

SD 

SEM 

CQ11 1 

64 

71.33 

24.74 

3.09 

CQ10 3 

64 

77.89 

18.64 

2.33 

Difference 

64 

-6.56 

31.10 

3.89 


95% upper bound for mean difference: -0.07. 
f Test of mean difference = 0 (vs. < 0): f = -1.69, p = .048. 

Table 10: Paired tforCQ13_3 and CQ14_1 



N 

M 

SD 

SEM 

CQ13 3 

64 

61.29 

23.90 

2.99 

CQ14 1 

64 

62.50 

27.11 

3.39 

Difference 

64 

-1.21 

40.65 

5.08 


95% upper bound for mean difference: 7.27. 
t Test of mean difference = 0 (vs. < 0): f = -0.24, p = .406. 

Table 11: Paired t for CQ21_1 and CQ22_2 



N 

M 

SD 

SEM 

CQ21 1 

62 

59.44 

31.71 

4.03 

CQ22 2 

62 

71.45 

27.52 

3.50 

Difference 

62 

-12.02 

40.81 

5.18 


95% upper bound for mean difference: -3.36. 
t Test of mean difference = 0 (vs. < 0): f = -2.32, p = .012. 

Table 12: Paired t for CQ6_3 and CQ7_1 



N 

M 

SD 

SEM 

CQ6 3 

68 

59.19 

25.95 

3.15 

CQ7 1 

68 

67.68 

26.57 

3.22 

Difference 

68 

-8.49 

32.54 

3.95 


95% Cl for mean difference: (-16.37, -0.62). 


t Test of mean difference = 0 (vs. not = 0): f = -2.15, p = .035. 
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Table 13: Paired t for CQ6_3 and CQ7_2 



N 

M 

SD 

SEM 

CQ6 3 

71 

58.06 

25.45 

3.02 

CQ7 2 

71 

65.60 

31.19 

3.70 

Difference 

71 

-7.54 

37.39 

4.44 


95% Cl for mean difference: (-16.39, 1.32). 
t Test of mean difference = 0 (vs. not = 0): f = -1 .70, p = .094 

Table 14: Paired t for CQ7_1 and CQ7_2 



N 

M 

SD 

SEM 

CQ7 1 

65 

66.88 

27.73 

3.44 

CQ7 2 

65 

68.31 

30.76 

3.81 

Difference 

65 

-1.42 

42.35 

5.25 


95% Cl for mean difference: (-11.92, 9.07). 


t Test of mean difference = 0 (vs. not = 0): f = -0.27, p = .787 
Table 15: Paired t for CQ8_3 and CQ21_3 



N 

M 

SD 

SEM 

CQ8 3 

95 

52.47 

29.03 

2.98 

CQ21 3 

95 

62.05 

24.70 

2.53 

Difference 

95 

-9.58 

28.15 

2.89 


95% upper bound for mean difference: -4.78. 
t Test of mean difference = 0 (vs. < 0): f = -3.32, p = .001. 

Table 16: Paired t for CQ21_1 and CQ12_3 



N 

M 

SD 

SEM 

CQ21 1 

63 

56.23 

31.55 

3.97 

CQ12 3 

63 

69.48 

23.18 

2.92 

Difference 

63 

-13.25 

38.23 

4.82 


95% upper bound for mean difference: -5.21. 
t Test of mean difference = 0 (vs. < 0): f = -2.75, p = .004. 

Table 17: Paired t for CQ14_2 and CQ14_3 



N 

M 

SD 

SEM 

CQ14 2 

66 

64.70 

23.74 

2.92 

CQ14 3 

66 

70.68 

22.74 

2.80 

Difference 

66 

-5.98 

33.04 

4.07 


95% upper bound for mean difference: 0.80. 
t Test of mean difference = 0 (vs. < 0): f = -1.47, p = .073. 

Table 18: Paired t for CQ21_1 and CQ22_1 



N 

M 

SD 

SEM 

CQ21 1 

76 

59.67 

30.96 

3.55 

CQ22 1 

76 

68.22 

29.78 

3.42 

Difference 

76 

-8.55 

21.03 

2.41 


95% upper bound for mean difference: -4.53. 
t Test of mean difference = 0 (vs. < 0): f = -3.54, p = .000. 


Table 19: Paired t for CQ6_2 and CQ6_1 



N 

M 

SD 

SEM 

CQ6 2 

64 

69.49 

27.99 

3.50 

CQ6 1 

64 

71.84 

26.10 

3.26 

Difference 

64 

-2.34 

39.06 

4.88 


95% upper bound for mean difference: 5.81. 
t Test of mean difference = 0 (vs. < 0): f = -0.48, p = .316. 


























































Table 20: Paired t for CQ11_3 and CQ11_2 



N 

M 

SD 

SEM 

CQ11 3 

66 

78.94 

19.71 

2.43 

CQ11 2 

66 

74.66 

22.85 

2.81 

Difference 

66 

4.28 

27.92 

3.44 


95% upper bound for mean difference: 10.02. 
f Test of mean difference = 0 (vs. < 0): f = 1.25, p = .891. 

Table 21: Paired t for CQ12_ 1 and CQ11_2 



N 

M 

SD 

SEM 

CQ12 1 

61 

72.05 

23.07 

2.95 

CQ11 2 

61 

78.07 

21.63 

2.77 

Difference 

61 

-6.02 

33.32 

4.27 


95% upper bound for mean difference: 1.10. 
f Test of mean difference = 0 (vs. < 0): f = -1.41, p = .082 

Table 22: Paired t for CQ11_3 and CQ12_1 



N 

M 

SD 

SEM 

CQ11 3 

64 

79.02 

18.89 

2.36 

CQ12 1 

64 

70.63 

22.49 

2.81 

Difference 

64 

8.40 

29.75 

3.72 


95% upper bound for mean difference: 14.61. 
t Test of mean difference = 0 (vs. < 0): f = 2.26, p = .986. 

Table 23: Paired t for CQ12_2 and CQ6_1 



N 

M 

SD 

SEM 

CQ12 2 

64 

70.47 

24.33 

3.04 

CQ6 1 

64 

72.42 

25.15 

3.14 

Difference 

64 

-1.95 

36.16 

4.52 


95% upper bound for mean difference: 5.59. 
t Test of mean difference = 0 (vs. < 0): f = -0.43, p = .334 

Table 24: Paired t for CQ15_2 and CQ15_1 



N 

M 

SD 

SEM 

CQ15 2 

62 

79.72 

16.49 

2.09 

CQ15 1 

62 

72.18 

22.95 

2.91 

Difference 

62 

7.54 

27.64 

3.51 


95% upper bound for mean difference: 13.40. 
t Test of mean difference = 0 (vs. < 0): f = 2.15, p = .982. 

Table 25: Paired t for CQ9_2 and CQ9_1 



N 

M 

SD 

SEM 

CQ9 2 

64 

64.96 

30.50 

3.81 

CQ9 1 

64 

70.16 

28.95 

3.62 

Difference 

64 

-5.20 

42.94 

5.37 


95% upper bound for mean difference: 3.77. 
t Test of mean difference = 0 (vs. < 0): f = -0.97, p = .168 

Table 26: Paired t for CQ10_1 and CQ9 3 



N 

M 

SD 

SEM 

CQ10 1 

66 

42.20 

31.97 

3.93 

CQ9 3 

66 

47.61 

26.95 

3.32 

Difference 

66 

-5.42 

43.86 

5.40 


95% upper bound for mean difference: 3.59. 


t Test of mean difference = 0 (vs. < 0): f = -1.00, p = .160 


























































Appendix P Descriptive Statisticai Resuits for Aiert 
Originator Surveys 


In Table 27, the variable is the question identifier from the Qualtrics survey. For example, 
S1Q5_1 represents Survey 1, Question 5, Subquestion 1. 


Table 27: Results of the Original Survey Ordinal Responses 


Variable 

N 

N* 

Min 

Q1 

Mdn 

Q3 

Max 

S1Q5 1 

18 

0 

34 

1,184 

59,000 

555,000 

5,000,000 

S1Q5 2 

14 

4 

0 

50 

25,000 

142,500 

10,000,000 

S1Q6 

18 

0 

1.0000 

2.0000 

2.0000 

2.0000 

2.0000 

S1Q7 1 

18 

0 

4.000 

5.000 

7.000 

7.000 

7.000 

S1Q7 2 

17 

1 

4.000 

5.000 

6.000 

7.000 

7.000 

S1Q7 3 

17 

1 

2.000 

4.500 

5.000 

7.000 

7.000 

S1Q7 4 

16 

2 

1.000 

4.000 

6.500 

7.000 

7.000 

S1Q8 1 

16 

2 

1.000 

4.000 

5.500 

7.000 

7.000 

S1Q8 2 

16 

2 

2.000 

5.000 

6.500 

7.000 

7.000 

S1Q8 3 

16 

2 

3.000 

6.000 

7.000 

7.000 

7.000 

S1Q8 4 

16 

2 

4.000 

7.000 

7.000 

7.000 

7.000 

S1Q9 1 

16 

2 

4.000 

5.000 

7.000 

7.000 

7.000 

S1Q9 2 

16 

2 

4.000 

6.000 

7.000 

7.000 

7.000 

S1Q9 3 

16 

2 

4.000 

6.000 

7.000 

7.000 

7.000 

S1Q10 1 

16 

2 

4.000 

6.000 

7.000 

7.000 

7.000 

S1Q10 2 

16 

2 

3.000 

4.000 

5.500 

6.000 

7.000 

S1Q10 3 

16 

2 

1.000 

2.250 

4.000 

5.750 

6.000 

S1Q10 4 

16 

2 

1.000 

1.000 

3.000 

4.750 

6.000 

S1Q11 1 

16 

2 

4.000 

4.000 

5.500 

6.000 

7.000 

S1Q11 2 

16 

2 

4.000 

6.000 

7.000 

7.000 

7.000 

S1Q12 1 

16 

2 

4.000 

6.000 

7.000 

7.000 

7.000 

S1Q12 2 

15 

3 

1.000 

4.000 

5.000 

6.000 

7.000 

S1Q12 3 

16 

2 

1.000 

2.250 

3.500 

5.000 

7.000 

S1Q13 1 

16 

2 

1.000 

1.000 

3.500 

4.000 

6.000 

S1Q13 2 

16 

2 

1.000 

3.250 

4.000 

5.000 

6.000 

S1Q13 3 

16 

2 

1.000 

4.000 

5.000 

6.000 

7.000 

S1Q13 4 

16 

2 

1.000 

4.250 

6.000 

7.000 

7.000 

S1Q13 5 

16 

2 

4.000 

6.000 

7.000 

7.000 

7.000 

S1Q14 1 

16 

2 

1.000 

3.000 

4.000 

5.000 

7.000 

S1Q14 2 

16 

2 

4.000 

4.000 

5.000 

6.000 

7.000 

S1Q14 3 

15 

3 

4.000 

6.000 

6.000 

7.000 

7.000 

S1Q15 1 

16 

2 

4.000 

5.250 

7.000 

7.000 

7.000 

S1Q15 2 

16 

2 

4.000 

5.000 

6.500 

7.000 

7.000 

S1Q15 3 

16 

2 

3.000 

5.000 

5.500 

6.000 

7.000 

S1Q15 4 

16 

2 

2.000 

3.250 

4.500 

5.000 

7.000 

S1Q16 1 

16 

2 

1.000 

3.000 

4.000 

4.000 

7.000 

S1Q16 2 

16 

2 

4.000 

4.000 

5.500 

6.000 

7.000 

S1Q16 3 

16 

2 

2.000 

4.000 

4.000 

6.750 

7.000 

S2Q5 1 

30 

1 

86 

3,825 

12,000 

208,750 

1,547,000 

S2Q5 2 

23 

8 

0 

100 

900 

20,000 

100,000 

S2Q6 

27 

4 

1.0000 

2.0000 

2.0000 

2.0000 

2.0000 
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Variable 

N 

N* 

Min 

Q1 

Mdn 

Q3 

Max 

S2Q7 1 

31 

0 

5.0000 

7.0000 

7.0000 

7.0000 

7.0000 

S2Q7 2 

31 

0 

4.000 

6.000 

7.000 

7.000 

7.000 

S2Q7 3 

31 

0 

1.000 

4.000 

5.000 

6.000 

7.000 

S2Q7 4 

31 

0 

1.000 

5.000 

7.000 

7.000 

7.000 

S2Q7 5 

31 

0 

1.000 

5.000 

5.000 

7.000 

7.000 

S2Q7 6 

31 

0 

1.000 

3.000 

5.000 

6.000 

7.000 

S2Q8 1 

31 

0 

3.000 

7.000 

7.000 

7.000 

7.000 

S2Q8 2 

31 

0 

3.000 

7.000 

7.000 

7.000 

7.000 

S2Q8 3 

31 

0 

1.000 

6.000 

7.000 

7.000 

7.000 

S2Q9 1 

31 

0 

4.000 

6.000 

7.000 

7.000 

7.000 

S2Q9 2 

31 

0 

1.000 

6.000 

7.000 

7.000 

7.000 

S2Q9 3 

31 

0 

1.000 

4.000 

6.000 

7.000 

7.000 

S2Q10 1 

30 

1 

4.000 

6.000 

7.000 

7.000 

7.000 

S2Q10 2 

31 

0 

2.000 

5.000 

5.000 

6.000 

7.000 

S2Q10 3 

31 

0 

1.000 

2.000 

4.000 

5.000 

7.000 

S2Q10 4 

31 

0 

1.000 

1.000 

3.000 

4.000 

7.000 

S2Q11 1 

31 

0 

1.000 

6.000 

7.000 

7.000 

7.000 

S2Q11 2 

31 

0 

1.000 

4.000 

6.000 

7.000 

7.000 

S2Q12 1 

31 

0 

1.000 

5.000 

6.000 

7.000 

7.000 

S2Q12 2 

31 

0 

1.000 

5.000 

6.000 

7.000 

7.000 

S2Q12 3 

31 

0 

1.000 

4.000 

4.000 

6.000 

7.000 

S2Q13 1 

31 

0 

1.000 

3.000 

4.000 

6.000 

7.000 

S2Q13 2 

31 

0 

1.000 

6.000 

7.000 

7.000 

7.000 

S2Q13 3 

31 

0 

1.000 

2.000 

4.000 

6.000 

7.000 

S2Q14 1 

31 

0 

1.000 

6.000 

7.000 

7.000 

7.000 

S2Q14 2 

31 

0 

1.000 

6.000 

7.000 

7.000 

7.000 

S2Q14 3 

31 

0 

1.000 

4.000 

6.000 

7.000 

7.000 

S2Q14 4 

31 

0 

1.000 

3.000 

5.000 

6.000 

7.000 

S2Q15 1 

31 

0 

1.000 

4.000 

5.000 

6.000 

7.000 

S2Q15 2 

31 

0 

1.000 

5.000 

6.000 

7.000 

7.000 

S2Q15 3 

31 

0 

1.000 

6.000 

7.000 

7.000 

7.000 

S2Q15 4 

30 

1 

1.000 

6.000 

7.000 

7.000 

7.000 

S2Q16 1 

31 

0 

1.000 

6.000 

7.000 

7.000 

7.000 

S2Q16 2 

31 

0 

1.000 

4.000 

6.000 

7.000 

7.000 

S3Q5 1 

17 

0 

1 

4,050 

225,000 

695,000 

4,000,000 

S3Q5 2 

13 

4 

2 

750 

3,000 

65,000 

500,000 

S3Q6 

17 

0 

1.0000 

2.0000 

2.0000 

2.0000 

2.0000 

S3Q7 1 

17 

0 

1.000 

3.000 

4.000 

4.500 

7.000 

S3Q7 2 

17 

0 

3.000 

4.000 

5.000 

5.500 

7.000 

S3Q7 3 

17 

0 

4.000 

5.500 

6.000 

6.000 

7.000 

S3Q7 4 

17 

0 

4.000 

6.500 

7.000 

7.000 

7.000 

S3Q8 1 

16 

1 

3.000 

4.000 

5.500 

7.000 

7.000 

S3Q8 2 

16 

1 

4.000 

5.000 

6.000 

7.000 

7.000 

S3Q8 3 

16 

1 

4.000 

6.000 

6.500 

7.000 

7.000 

S3Q8 4 

17 

0 

4.000 

6.000 

7.000 

7.000 

7.000 

S3Q9 1 

16 

1 

5.000 

7.000 

7.000 

7.000 

7.000 

S3Q9 2 

16 

1 

5.000 

6.250 

7.000 

7.000 

7.000 

S3Q9 3 

17 

0 

4.000 

6.000 

7.000 

7.000 

7.000 

S3Q10 1 

17 

0 

5.000 

6.500 

7.000 

7.000 

7.000 

S3Q10_2 

17 

0 

3.000 

5.000 

6.000 

7.000 

7.000 
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Variable 

N 

N* 

Min 

Q1 

Mdn 

Q3 

Max 

S3Q10 3 

17 

0 

1.000 

3.000 

4.000 

4.500 

6.000 

S3Q10 4 

16 

1 

1.000 

2.000 

3.000 

3.750 

5.000 

S3Q11 1 

17 

0 

3.000 

5.500 

7.000 

7.000 

7.000 

S3Q11 2 

17 

0 

4.000 

6.000 

6.000 

7.000 

7.000 

S3Q11 3 

17 

0 

3.000 

4.500 

5.000 

7.000 

7.000 

S3Q12 1 

16 

1 

4.000 

6.000 

7.000 

7.000 

7.000 

S3Q12 2 

16 

1 

3.000 

6.000 

6.000 

7.000 

7.000 

S3Q12 3 

16 

1 

1.000 

3.000 

5.000 

6.000 

6.000 

S3Q12 4 

16 

1 

1.000 

2.000 

4.000 

5.000 

6.000 

S3Q13 1 

16 

1 

4.000 

5.000 

6.000 

6.000 

7.000 

S3Q13 2 

16 

1 

4.000 

6.000 

6.000 

7.000 

7.000 

S3Q13 3 

16 

1 

5.000 

7.000 

7.000 

7.000 

7.000 

S3Q14 1 

16 

1 

1.000 

4.500 

6.500 

7.000 

7.000 

S3Q14 2 

16 

1 

1.000 

5.000 

6.000 

7.000 

7.000 

S3Q14 3 

16 

1 

3.000 

4.250 

5.000 

6.750 

7.000 

S3Q14 4 

16 

1 

2.000 

3.250 

4.500 

6.000 

7.000 

S3Q15 1 

16 

1 

3.000 

6.250 

7.000 

7.000 

7.000 

S3Q15 2 

16 

1 

4.000 

6.000 

6.000 

7.000 

7.000 

S3Q15 3 

16 

1 

1.000 

2.250 

3.500 

4.000 

6.000 

S3Q16 1 

16 

1 

3.000 

6.000 

7.000 

7.000 

7.000 

S3Q16 2 

16 

1 

3.000 

6.250 

7.000 

7.000 

7.000 

S3Q16_3 

16 

1 

3.000 

7.000 

7.000 

7.000 

7.000 
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Appendix Q Comparative Anaiysis of Paired Questions for 
Aiert Originator Surveys 


Table 28: Paired t for S1CQ7_ 1 and S1CQ7_2 



N 

M 

SD 

SEM 

S1CQ7 1 

17 

83.68 

19.18 

4.65 

S1CQ7 2 

17 

84.71 

16.58 

4.02 

Difference 

17 

-1.03 

19.29 

4.68 


95% Cl for mean difference: [-10.94, 8.89], 
t Test of mean difference = 0 (vs. not = 0): f = -0.22, p = .829. 

Table 29: Paired t for S1CQ7_ 1 and S1CQ7_3 



N 

M 

SD 

SEM 

S1CQ7 1 

17 

83.68 

19.18 

4.65 

S1CQ7 3 

17 

76.03 

24.92 

6.04 

Difference 

17 

7.65 

29.65 

7.19 


95% lower bound for mean difference: -4.91. 
t Test of mean difference = 0 (vs. > 0): f = 1.06, p = .152. 

Table 30: Paired t for S1CQ7_1 and S1CQ7_4 



N 

M 

SD 

SEM 

S1CQ7 1 

16 

84.53 

19.48 

4.87 

S1CQ7 4 

16 

70.78 

33.67 

8.42 

Difference 

16 

13.8 

40.8 

10.2 


95% lower bound for mean difference: -4.1. 
f Test of mean difference = 0 (vs. > 0): f = 1.35, p = .099. 

Table 31: Paired t for SICQ8_1 and S1CQ8__2 



N 

M 

SD 

SEM 

S1CQ8 1 

16 

72.66 

29.26 

7.32 

S1CQ8 2 

16 

80.94 

24.71 

6.18 

Difference 

16 

-8.28 

11.32 

2.83 


95% upper bound for mean difference: -3.32. 
t Test of mean difference = 0 (vs. < 0): f = -2.93, p = .005. 

Table 32: Paired t for SICQ8_1 and S1CQ8_3 



N 

M 

SD 

SEM 

S1CQ8 1 

16 

72.66 

29.26 

7.32 

S1CQ8 3 

16 

85.47 

21.68 

5.42 

Difference 

16 

-12.81 

15.62 

3.91 


95% upper bound for mean difference: -5.97. 
t Test of mean difference = 0 (vs. < 0): f = -3.28, p = .003. 

Table 33: Paired t for S1CQ8_ 1 and S1CQ8_4 



N 

M 

SD 

SEM 

S1CQ8 1 

16 

72.66 

29.26 

7.32 

S1CQ8 4 

16 

93.28 

12.03 

3.01 

Difference 

16 

-20.63 

27.76 

6.94 


95% upper bound for mean difference: -8.46. 
t Test of mean difference = 0 (vs. <0): t= -2.97, p = .005. 
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Table 34: Paired t for S2CQ8_1 and S2CQ8_2 



N 

M 

SD 

SEM 

S2CQ8 1 

31 

94.35 

12.31 

2.21 

S2CQ8 2 

31 

92.90 

13.68 

2.46 

Difference 

31 

1.452 

4.645 

0.834 


95% Cl for mean difference: [-0.252, 3.155], 
t Test of mean difference = 0 (vs. not = 0): f = 1.74, p = .092. 

Table 35: Paired t for S2CQ8_ 1 and S2CQ8_3 



N 

M 

SD 

SEM 

S2CQ8 1 

31 

94.35 

12.31 

2.21 

S2CQ8 3 

31 

81.85 

30.61 

5.50 

Difference 

31 

12.50 

26.92 

4.83 


95% lower bound for mean difference: 4.29. 
t Test of mean difference = 0 (vs. > 0): f = 2.59, p = .007. 

Table 36: Paired i for S3CQ8_1 and S3CQ8_4 



N 

M 

SD 

SEM 

S3CQ8 1 

16 

74.38 

22.50 

5.63 

S3CQ8 4 

16 

87.97 

16.56 

4.14 

Difference 

16 

-13.59 

28.68 

7.17 


95% upper bound for mean difference: -1.02. 
t Test of mean difference = 0 (vs. < 0): f = -1.90, p = .039. 

Table 37: Paired t for S1CQ9_1 and S1CQ9_3 



N 

M 

SD 

SEM 

S1CQ9 1 

16 

86.41 

15.30 

3.83 

S1CQ9 3 

16 

90.94 

13.22 

3.31 

Difference 

16 

-4.5 

38.72 

2.18 


95% upper bound for mean difference: -0.71. 
t Test of mean difference = 0 (vs. < 0): f = -2.08, p = .028. 

Table 38: Paired i for S2CQ9_1 and S2CQ9_3 



N 

M 

SD 

SEM 

S2CQ9 1 

31 

89.19 

14.99 

2.69 

S2CQ9 3 

31 

71.21 

33.28 

5.98 

Difference 

31 

17.98 

31.12 

5.59 


95% lower bound for mean difference: 8.50. 
t Test of mean difference = 0 (vs. > 0): f = 3.22, p = .002. 

Table 39: Paired t for S3CQ9_1 and S3CQ9_3 



N 

M 

SD 

SEM 

S3CQ9 1 

16 

93.44 

9.48 

2.37 

S3CQ9 3 

16 

88.59 

16.73 

4.18 

Difference 

16 

4.84 

12.60 

3.15 


95% lower bound for mean difference: -0.68. 
t Test of mean difference = 0 (vs. > 0): f = 1.54, p = .072. 

Table 40: Paired t for S3CQ8_1 and S3CQ8_4 



N 

M 

SD 

SEM 

S3CQ8 1 

16 

74.38 

22.50 

5.63 

S3CQ8 4 

16 

87.97 

16.56 

4.14 

Difference 

16 

-13.59 

28.68 

7.17 


95% upper bound for mean difference: -1.02. 
t Test of mean difference = 0 (vs. < 0): f = -1.90, p = .039. 
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Table 41: Paired t for S1CQ11_1 and S1CQ11_2 



N 

M 

SD 

SEM 

S1CQ11 1 

16 

73.75 

18.48 

4.62 

S1CQ11 2 

16 

90.78 

11.93 

2.98 

Difference 

16 

-17.03 

17.13 

4.28 


95% upper bound for mean difference: -9.52. 
t Test of mean difference = 0 (vs. < 0): t= -3.98, p = .001 

Table 42: Paired t for S3CQ11_ 1 and S3CQ11_3 



N 

M 

SD 

SEM 

S3CQ11 1 

17 

86.91 

20.19 

4.90 

S3CQ11 3 

17 

76.47 

21.92 

5.32 

Difference 

17 

10.44 

27.62 

6.70 


95% lower bound for mean difference: -1.25. 
t Test of mean difference = 0 (vs. > 0): f = 1.56, p = .069. 

Table 43: Paired t for S1CQ12_1 and S1CQ12_3 



N 

M 

SD 

SEM 

S1CQ12 1 

16 

89.69 

13.00 

3.25 

S1CQ12 3 

16 

44.69 

30.26 

7.57 

Difference 

16 

45.00 

33.22 

8.30 


95% lower bound for mean difference: 30.44. 
t Test of mean difference = 0 (vs. > 0): f = 5.42, p = .000. 

Table 44: Paired t for S2CQ12_1 and S2CQ12_3 



N 

M 

SD 

SEM 

S2CQ12 1 

31 

77.42 

29.03 

5.21 

S2CQ12 3 

31 

58.39 

30.53 

5.48 

Difference 

31 

19.03 

36.38 

6.53 


95% lower bound for mean difference: 7.94. 
t Test of mean difference = 0 (vs. > 0): f = 2.91, p = .003. 

Table 45: Paired t for S3CQ12_1 and S3CQ12_4 



N 

M 

SD 

SEM 

S3CQ12 1 

16 

90.31 

13.13 

3.28 

S3CQ12 4 

16 

43.59 

30.70 

7.67 

Difference 

16 

46.72 

27.41 

6.85 


95% lower bound for mean difference: 34.70. 
t Test of mean difference = 0 (vs. > 0): f = 6.82, p = .000. 

Table 46: Paired t for S3CQ13_1 and S3CQ13_2 



N 

M 

SD 

SEM 

S3CQ13 1 

16 

79.06 

14.14 

3.53 

S3CQ13 2 

16 

87.81 

12.41 

3.10 

Difference 

16 

-8.75 

12.65 

3.16 


95% upper bound for mean difference: -3.21. 
t Test of mean difference = 0 (vs. < 0): f = -211 , p = .007 

Table 47: Paired t for S2CQ13_3 and S2CQ13_2 



N 

M 

SD 

SEM 

S2CQ13 3 

3 

146.85 

35.46 

6.37 

S2CQ13 2 

31 

90.00 

17.45 

3.13 

Difference 

31 

-43.15 

33.20 

5.96 


95% upper bound for mean difference: -33.02. 
t Test of mean difference = 0 (vs. < 0): f = -7.23, p = .000 


























































Table 48: Paired t for S1CQ14_1 and S1CQ14_3 



N 

M 

SD 

SEM 

S1CQ14 1 

15 

50.00 

27.42 

7.08 

S1CQ14 3 

15 

87.33 

13.93 

3.60 

Difference 

15 

-37.33 

29.39 

7.59 


95% upper bound for mean difference: -23.97. 
t Test of mean difference = 0 (vs. < 0): f = -4.92, p = .000 

Table 49: Paired t for S2CQ15_1 and S2CQ15_4 



N 

M 

SD 

SEM 

S2CQ15 1 

30 

64.08 

23.87 

4.36 

S2CQ15 4 

30 

89.92 

18.63 

3.40 

Difference 

30 

-25.83 

21.50 

3.93 


95% upper bound for mean difference: -19.16. 
t Test of mean difference = 0 (vs. < 0): f = -6.58, p = .000 

Table 50: Paired t for S3CQ15_1 and S3CQ15_3 



N 

M 

SD 

SEM 

S3CQ15 1 

16 

89.06 

19.79 

4.95 

S3CQ15 3 

16 

39.06 

25.59 

6.40 

Difference 

16 

50.00 

35.92 

8.98 


95% lower bound for mean difference: 34.26. 
t Test of mean difference = 0 (vs. > 0): f = 5.57, p = .000. 

Table 51: Pairedtfor S1CQ16_1 and S1CQ16_3 



N 

M 

SD 

SEM 

S1CQ16 1 

16 

47.50 

28.91 

7.23 

S1CQ16 3 

16 

62.97 

27.99 

7.00 

Difference 

16 

-15.47 

35.49 

8.87 


95% upper bound for mean difference: 0.08. 
f Test of mean difference = 0 (vs. < 0): f = -1.74, p = .051 

Table 52: Paired t for S2CQ16_1 and S2CQ16_2 



N 

M 

SD 

SEM 

S2CQ16 1 

31 

90.08 

19.09 

3.43 

S2CQ16 2 

31 

74.03 

28.17 

5.06 

Difference 

31 

16.05 

22.55 

4.05 


95% lower bound for mean difference: 9.17. 
t Test of mean difference = 0 (vs. > 0): f = 3.96, p = .000. 

Table 53: Paired t for S3CQ16_1 and S3CQ16_3 



N 

M 

SD 

SEM 

S3CQ16 1 

16 

84.84 

22.67 

5.67 

S3CQ16 3 

16 

92.03 

16.89 

4.22 

Difference 

16 

-7.19 

17.67 

4.42 


95% upper bound for mean difference: 0.56. 
t Test of mean difference = 0 (vs. < 0): f = -1.63, p = .062 


















































Table 54: Two-Sample t Test for S2CQ7_1 vs. S1CQ7_4 


N 

M 

SD 

SE 

M 

S2CQ7 1 

31 

96.29 

5.20 

0.93 

S1CQ7_4 

16 

70.8 

33.7 

8.4 


Difference = |j (S2CQ7_1) - |j (S1CQ7_4) 

Estimate for difference: 25.51 
95% Cl for difference: (7.46, 43.56) 

f Test of difference = 0 (vs. not =): f = 3.01, p = .009, df=^5 
Table 55: Two-Sample t Test for S2CQ71 vs. S2CQ7 6 


N 

M 

SD 

SE 

M 

S2CQ7 1 

31 

96.29 

5.20 

0.93 

S2CQ7_6 

31 

61.3 

30.4 

5.5 


Difference = p (S2CQ7_1) - p (S2CQ7_6) 

Estimate for difference: 35.00 

95% Cl for difference: (23.69, 46.31) 

f Test of difference = 0 (vs. not =): f = 6.31, p = .000, df= 31 

Table 56: Two-Sample t Test for S2CQ7_1 vs. S2CQ7_5 


N 

M 

SD 

SE 

M 

S2CQ7 1 

31 

96.29 

5.20 

0.93 

S2CQ7_5 

31 

74.8 

24.1 

4.3 


Difference = p (S2CQ7_1) - p (S2CQ7_5) 

Estimate for difference: 21.53 

95% Cl for difference: (12.50, 30.57) 

t Test of difference = 0 (vs. not =): t = 4.85, p = .000, df= 32 

Table 57: Two-Sample t Test for S2CQ71 vs. S2CQ7 3 


N 

M 

SD 

SE 

M 

S2CQ7 1 

31 

96.29 

5.20 

0.93 

S2CQ7_3 

31 

69.0 

25.9 

4.6 


Difference = p (S2CQ7_1) - p (S2CQ7_3) 

Estimate for difference: 27.26 

95% Cl for difference: (17.61,36.91) 

f Test of difference = 0 (vs. not =): f = 5.75, p = .000, df= 32 

Table 58: Two-Sample t Test for S2CQ7_1 vs. S2CQ7_2 


N 

M 

SD 

SE 

M 

S2CQ7 1 

31 

96.29 

5.20 

0.93 

S2CQ7_2 

31 

91.6 

10.8 

1.9 


Difference = p (S2CQ7_1) - p (S2CQ7_2) 

Estimate for difference: 4.68 
95% Cl for difference: (0.34, 9.02) 

t Test of difference = 0 (vs. not =): f = 2.17, p = .035, df= 43 
Table 59: Two-Sample t Test for S3CQ7_1 vs. S3CQ7_4 


N 

M 

SD 

SE 

M 

S3CQ7 1 

17 

48.2 

25.6 

6.2 

S3CQ7_4 

17 

88.5 

18.5 

4.5 


Difference = p (S3CQ7_1) - p (S3CQ7_4) 

Estimate for difference: -40.29 

95% Cl for difference: (-55.99, -24.60) 

t Test of difference = 0 (vs. not =): t = -5.25, p = .000, df= 29 
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Table 60: Two-Sample t Test for S3CQ7_1 vs. S3CQ7_3 


N 

M 

SD 

S£ 

M 

S3CQ7 1 

17 

48.2 

25.6 

6.2 

S3CQ7_3 

17 

81.6 

16.3 

3.9 


Difference = |j (S3CQ7_1) - |j (S3CQ7_3) 

Estimate for difference: -33.38 

95% Cl for difference: (-48.50, -18.27) 

t Test of difference = 0 (vs. not =): t = -4.53, p = .000, df= 27 

Table 61: Two-Sample t Test for S3CQ7_1 vs. S3CQ7_2 


N 

M 

SD 

SE 

M 

S3CQ7 1 

17 

48.2 

25.6 

6.2 

S3CQ7_2 

17 

64.7 

21.9 

5.3 


Difference = p (S3CQ7_1) - p (S3CQ7_2) 

Estimate for difference: -16.47 

95% Cl for difference: (-33.14, 0.20) 

t Test of difference = 0 (vs. not =): f = -2.02, p = .053, df = 31 
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Appendix R Alert Originator Survey Graphical Results 


Alert Originator Conglomeration 
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Figure 13: Histogram ofS1-3CQ10J, S1-3CQ10J2, S1-3CQ10J3, and S1-3CQ10_4 
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Figure 14: Histogram of S1-3CQ1514J, S1-CQ1514_2, S1-3CQ1514_3, and S1-3CQ1514_4 
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Histograms from Alert Originator Survey 1 
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Figure 15: Histogram ofS1CQ7_1, S1CQ7_2, S1CQ7_3, and S1CQ7_4 
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Figure 16: Histogram of SIQ8_1, S1Q8_2, S1Q8_3, and S1Q8_4 
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Figure 17: Histogram of SIQ9_1, S1Q9_2, and S1Q9_3 
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Figure 18: Histogram of S1Q10_1, S1Q10_2, S1Q10_3, and SIQ10_4 
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Figure 19: Histogram of SIQ11_1 and S1Q11_2 
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Figure 20: Histogram of S1Q12_1, S1Q12_2, and S1Q12_3 
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Figure 21: Histogram ofS1Q13_1, S1Q13_2, S1Q13_3, S1Q13_4, and S1Q13_5 
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Figure 22: Histogram of S1Q14_1, S1Q14_2, and S1Q14_3 
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Figure 23: Histogram ofS1Q15_1, S1Q15_2, S1Q15_3, and S1Q15_4 
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Figure 24: Histogram of S1Q16_1, S1Q16_2, and S1Q16_3 
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Histograms from Alert Originator Survey 2 
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Figure 25: Histogram ofS2Q7_1, S2Q7_2, S2Q7_3, S2Q7_4, S2Q7_5, and S2Q7_6 
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Figure 26: Histogram of S2Q8_1, S2Q8_2, and S2Q8_3 
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Figure 27: Histogram of S2Q9_1, S2Q9_2, and S2Q9_3 
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Figure 28: Histogram of S2Q10_1, S2Q10_2, S2Q10_3, and S2Q10_4 
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Figure 29: Histogram of S2Q11_1 and S2Q11J2 
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Figure 30: Histogram of S2Q12_1, S2Q12_2, and S2Q12_3 
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Figure 31: Histogram of S2Q13_1, S2Q13_2, and S2Q13_3 
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Figure 32: Histogram ofS2Q14_1, S2Q14_2, S2Q14_3, and S2Q14_4 


CMU/SEI-2013-SR-021 | 182 










































































































S2Q 15_1 

Mean 

4.806 

StDev 

1.376 

N 

31 

S2Q15 2 

Mean 

5.774 

StDev 

1.334 

N 

31 

S2Q15 3 

Mean 

6.258 

StDev 

1.264 

N 

31 

S2Q15 4 

Mean 

6.433 

StDev 

1.223 

N 

30 



Figure 33: Histogram ofS2Q15_1, S2Q15_2, S2Q15_3, and S2Q15_4 
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Figure 34: Histogram of S2Q16_1 and S2Q16_2 
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Histograms from Alert Originator Survey 3 
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Figure 35: Histogram ofS3Q7_1, S3Q7_2, S3Q7_3, and S3Q7_4 
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Figure 36: Histogram of S3Q8_1, S3Q8_2, S3Q8_3, and S3Q8_4 


CMU/SEI-2013-SR-021 | 184 













































































































S3Q9_1 

Mean 

6.688 

StDev 

0.7042 

N 

16 

S3Q9_2 

Mean 

6.625 

StDev 

0.7188 

N 

16 

S3Q9_3 

Mean 

6.412 

StDev 

1.064 

N 

17 



Figure 37: Histogram ofS3Q9_1, S3Q9_2, and S3Q9_3 
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Figure 38: Histogram of S3Q10_1, S3Q10_2, S3Q10_3, and S3Q10_4 
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Figure 39: Histogram of S3Q11_1, S3Q11_2, and S3Q11_3 
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Figure 40: Histogram of S3Q12_1, S3Q12_2, S3Q12 3, and S3Q12_4 
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Figure 41: Histogram of S3Q13_1, S3Q13_2, S3Q13_3, and S3Q13_4 
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Figure 42: Histogram ofS3Q14_1, S3Q14_2, S3Q14_3, and S3Q14_4 
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Figure 43: Histogram of S3Q15_1, S3Q15_2, and S3Q15_3 
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Figure 44: Histogram of S3Q16_1, S3Q16_2, and S3Q16_3 
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Appendix S Public Trust Model Formula Extract from 
AgenaRisk 


<?itml •rtcoding*TJTF-8“?> 

<Ri9fc_Mod*l Numb*r_Of_Ri8k_Obj*cls=*1 *> 

<Unks_btlw»»n_Risk_Ob^s^ 

• <Ri8k_Obt*ct Ngmb»r_Of_Nedts='89* Shp«1_D#scnp<ion='N«w Risk Obisct' »d="N*w Risk Obj*ct_f'> 
<lnpul_Nbd*s/> 

^Outpul.NodssA- 

• <Nods td='MQ‘ Nsms^lOQ HMiring' Typs='Continuous lnttrvar^ 

<Parsnl_Nodt>M27^61</P«rffll_Nodt> 

<Stfll«s>-lrrfinity • O.OtgO 0 • lO.O^To.O • ln(inity:/St«lss:- 
<NPT/> 

<Ezprtssion Partition«d=*Nls*' S«nulatiQns'tius*>Anthmtlic((M37-»M61)/2)<.*Expr«s««n> 

<Ob*bivctiont/> 

■^od*> 

• <No<ls id=*M0_1'‘ N«m*-*181_Und«r3t«nding’ Typ*=*Cantinuout InMrvtf*:- 

<Parsnl_Nods>M0®M90^91<^«rsnt_N(>dt> 

<Slatf sMnfinity • 0.0@0 0 • 10.0^10.0 • lnfiniy</St«lt$> 

<NPT/> 

•--Expression PertiUoned^'Telse* Siniulelien-*true*^‘Anihmet>c«MO/100>’((M904M91V2))‘'JCxpress»»n> 
<Obssrvslions/> 

<^ed#> 

• <Node id='M0 1 1* Name^'IOS Balisvtng' Type=*Centinueus Intervar - 

<Par»nrN»ds>M0_1lSM81®M92-:/Pafent.Nede’ 

<Sial«s>-ln5nny - 0.0@0 0 - 10.0^10.0 - lrAnily</Stales> 

<NPT/> 

^Expression Pariaioned=Talse‘ Simuiation=true‘>Anlhtnelic((M0_1/1O0)'((M8l4M92)/2))</Expression> 
<Observaiions/> 

<^6de> 

- <Noda id=^0 1 1 1*Naino=*103 Acting* Type=*Centinueus Intervar' 

<Par»nrN<>da>M0_1_1®M71QM75</Parani_No«a» 

<Stala$>-4nfSnity • 0.0^ 0 • 10.0^10.0 • lnAr^</Stalts> 

«:NPT/> 

<Expression Partaion«d='rals«' Siniulaiion=*lrua*>AnthmelK«MO_1_1/tOO>'((M71eM73)/2»<Cxpr«ss»on> 
<Obstrvalions/> 

<Aiode> 

- <Node id='M4* Nami='89.Typa_Of_Alo(t’ Ty|M=n.abolad'> 

<Piieni_Nedi > 

•^Slates>Presi<len(ial AlertQlmminent Threat^Amber Aler1<=/Stales> 

- <NPT> 

<NPT.Row>0.3M33334-:/t^T_Row> 

<NPT Row>0.33333334</I«>T R9W> 

<NPT’Row>0.33333334</?^T’Row> 

<iWPT> 

<Expression Paditioned^'falsa' SimulalionsTalsa'/^ 

<Qbservations/> 

■<^oda> 

• <Node id-144_1' Nanit=''7_rroguency‘ Type=*Condnuoiis Interval* => 

<Paienl_Nods > 

<Stalas>0.0 - S0.0@60.0- 100.0</Slatas> 

- «NPT> 

<NPT_Rovr>0-S</NPT_Rovr> 

<NPT_Rovr>0-S</NPT_Rovr> 

<yNPT> 

<Expra$$ien P8ridionad='ralsa’ Sanulatien»talae‘>Unifenn(0.100)<^pres$ien^ 

<Obsaivalions/> 

-'^ode> 

• -t<lode id-*M4_1_1* Nam#-*44_Redui>dancy_Of_Alerting' Type^'Contimiout IrMerval’:- 

<Paienl_Nods > 

<Stat«s>0.0-SO.O^O.O'100.l></Slaies> 

- <NPT> 

<NPT_Row>0-5-5/NPT_Row> 

<NPT Row>0.8</NPT Row> 

<>NPT> 

<Exprassion Par14ioned-*falta’ Smutation-talM‘>Unifonn(0.100)<£xpr«ssion> 

<Qbservaiions/> 

<Mbda> 

- -'Node id2iyi4_1_1_1* Nam«s*1S_Easy_Addl_Follow_Us_Mech«niefns' T^e='Centinuoiis Intervai'^ 

<Paienl Nods > 

<Stalas>0.0-SO.O@80.0- 100.l></^ates> 

• <rslPT> 

<NPT Row>0.S-:/?'#>T Row> 

<NPtIrow>0.6</NPtIrow> 

<yNPT> 

<Exprassion Pai14ion«d=*falaa’ Smulation='ralM*>Unifonn<0.100)<£xpr«ssion> 

<Obsaivaiions/> 

<^ode> 

• <Node Name=71_MossageJn_Pnmary_Language' T)f)e=*Continuous Interval'=- 

<Paienl_Nods > 

<Slales>0.0-S(>.0@60.0' 100.l></^ates> 

• <NPT> 

<NPT Row>0.5</NPT Row> 

<NPTIRow>0.S'^tIRow> 

<iNPT> 
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<Expr«ss*en Part<ion«d=Yal»*‘ Simulalion=Talsa'‘>Unif6rm(0.100)<€xprass^> 

<:Obsaivaiions/> 

<Moda> 

<Noda »d=‘’M4_1_1_2* Nania=*iS_Local_Juri»dictiona.Act.UncoordinBt*d‘ Typ«=‘CorMinueua Intarval*^ 

<Pai»nt_Nad« > 

<Stala$>0.0-$O.OQ60.0- 100.0<ySlatas> 

• <NPT» 

<hPT_R<w>0,5</?«>T_Row> 

<NPT Row>0.&</NPT Raw> 

</NPT> 

<Expra»sion Parltionad-’falaa’ Sanu(ation 27 al««*>Unifonn( 0 . 100 )<€xprasfion> 

<ObS*rv«iont/> 

<Mada> 

<f4o(le •d-*IIM_1_1.2.1* Nama»*21_Clarity_MMsage_Spalling_Grammar_Contant’ Typa^'Centinueut Intarval' - 
<Pai»nl N»dt > 

<Stala«>0.0'SO.Oe60.0' 100.l></Slatts> 

. <NPT> 

'-NPT_R«w>O.S-VNPT_R«w> 

<NPT_Row>0.5</NPT_R«w> 

<yNPT> 

cExprassion Parlitionad-'Yalaa’ Smu(aiion-talaa*>Unifonn(0.100)<£xprassion> 

<Obsaivationi/> 

<^eda> 

• <Noda id=HI4_1_1_2_1_1' Nama»*3_Public_Awaran«s5_of_WEA' Typa-'Continuous IntervaT? 

<Par»nl Nadi > 

>cSta)as~0.0*SO.Oe60.0- 100.0</Sla(as> 

• <NPT> 

<NPT Row>0.5</NPT R«w> 

<NPT’R«w>0.5</NPT’R«w> 

<iNPT> 

<Ezpra»ion Paddionad^triaa* Sanu(a(ions'U»«*>Unifonn<0,100)<^pr«sson> 

<Ob»»rvations/> 

<Mada> 

• <Nodi id=*M4_1^1_2_1_1_1* Namf="M_Hi»to»y_Of_Ral*vanca‘ Typa-‘Cofitinuou« kitarval‘> 

<Paitnl_Nodi » 

cStalas>0.0*50.0i9$0.0* 100.0</Sl«(»s> 

. <NPT> 

<NPT_Row>0.6</NPT_Row> 

<NPT Row>O.S</NPT Row> 

<INP1> 

<Expra«$ion PariilienadstalM’ Sanula(ien3tal$a'>Unifonn<0.100H^Pf*ssidA^ 

<Obsaivaiidna/> 

<^edt> 

<Noda id-'M4_1_2* Namas'd.Opt.Out.Rata* Typa^'Contiflueua Intarvar * 

<Paranl~rtoda>M28@M29^30~/Paren(_Noda> 

<Stalas>0.0-50.0@60.0- 100.0</Slatas> 

- <NPT> 

<NPT_Rori>09ei 0®! <>@1 0®l -0®1.0®1.0e0.73$842l®l.0®1 0®1 0®l J)®t ll®1.0®! 0®t .0®0.e8421054®1.0®! .0®0.736342l ®1. 

<NPT RcM>O.0e0.0®Oi)eO.I)®O.O®l).0eO.0®O.O®0 263l$79®O.O®O 0®O.0®OOeO9®O.0®l)0e0.O®O.3157894e®O.0®0.O®0 2631S79®0. 
</WPT> 

<Exprassion PaftitH>nad='falta* Sirnulatien=*falaa'>Anihmalic((M28aM294M30V3)<£xpfasM)n> 

<Obsaivaiions/» 

<Mpda' 

-^oda id=*M4_2* Nama=’49_Alafta_Viawad_At_Spam* Typa='Centinueut lnlarval‘> 

<Parani Noda>MW®M77®M93</Parart Noda» 

<Stalas>0.0-SO.O®60.0- 100 0</Sta(at> 

. <NPT> 

<NPT Rcrr 0.0®l 0®l 0®1 .0®1 0®l 0®1.9®1 .0®9.73e$421®t.0®I 0®1 0®l 0®l 0®t .0®! 0®t .0®0.ft8421054®1.0®1.0®0.7393421 ®t. 
<NPT RaM>00®0.0®00®0 0®0.0®9.0®0.9®O.OeO^S31579®0.0®0 0®0 0®00®00®0.0®90®0.0®0.31578946®0.0®O Oe0.2631579®0. 
</NPT> 

cExpraasion Partitionad^Talaa’ Siniulalion=Talaa'>Arithmalic((M93aM&44M77)/3)<Cxprassion> 

<Obsaivalions/> 

<Mada> 

■d4oda id='W0_1_1_2* Naina*'^7_Cofifirmat»n_By_So<ial_Madia' Typa-‘Continu«u* lntarvar=> 

«Parani_Nl)d4>M43</Parart_Nod4> 

<Stales>0.0-90.0®SO.O- tOlTo^/Slataa^ 

. <NPT> 

<NPT Row>0.0®1.0®0.4444444$<yNPT Row> 

<NPT~Row>0.0®0 0®0.$SSSSS€<MPT Row> 

<iNPT> 

cExprassion ParMionad-talsa' Smutalion=‘Talaa''>Ariihmalic(M43)</Expras3ion> 

<Obsarvaiion«/> 

<^oda> 

• <^4oda id='M4_1_1_2_1_1_1_1_1* Nama=*10_Aetjan_Ta_Take" Typa=*Cantinuoua IntarvaT - 

<Paianl_Nadi > 

<Stalas>0.0-50.0®&0.0- 100.0</Slatas> 

• <NPT> 

<NPT_R«w>0.5</NPT_Rgw» 

<NPT_Row>0.5-:/F'd>T_Row* 

«yNPT> 

cExprassion Padilionad-talaa* Smutation-*falaa*>Unifonn<0.100)<£xprassion> 

<Obsarvaiions/> 
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<Notf« *d=‘M4_1_1_2_1 Nani*=*12_AI«rt_Seurc*‘ Typ*=\*b«ll«d‘ • 

• <NPT> 

<NPT Row>0.33333334<>T^T Row> 

<NPT Row>0.33333334</T^T Row> 

■=NPT Row>0.33333334<>M>T Row* 

</NPT> 

<Expr»ssion PaftHionod-’falto* Si(nulation='fah«*/> 

<Obs*rvtfions/> 

<Modt> 

<t4odo id“'M4_1_1_2_1_1_1_1_1_2* Namt="2l_Whoro_To_G*_For_Mof*Jnfo' Typ«=*ContifHi««» Inlofvol** 

<Pai»nl Nod* > 

<S(«l*s>0.0'SO.Oe60.0- 100.0</Slatos> 

• <NPT> 

<NPT R«w>0.5</NPT R«w> 

'NPT”r«w>O.S</?^t”r«w> 

<4dPT> 

<Exf>r*Mion Patldionod^taloo* Smu<«tions'M«o'>Unifonn{0.100)<^pr«s«on> 

<Obsoivaiioni/> 

■c4<lodo> 

-tdodo *d=*W4_1_1_2_1_1^1_1_1_3* Namo-'3_Hiitery_Of.Final_Conin«jnication' Typ«='Continuou* Intorval*^- 

<Pirtnl_Nodi > 

cSt«os>0.0-S0.0@60.0- 100.0</Sla(*s> 

• <NPT> 

<NPT_Row>0-S</NPT_Row> 

<NPT Row>0.5</NPT Row* 

</NPT> 

<Expr*ssion Parlilionodsfolta* Simutation-tal*a*:'Unifonn<0.100)<€xpr«saion> 

<Oba*rvaiions/> 

■«^od#> 

<Noda •d=*lliU_1_1_2_1_1_1_1_1_4* Naiin*='7#_Why_Sh6uldJ_Act' Typo^’ContiiMJOu* Intarvar- 
<Par»nl Noda > 

<Sial*s>0.0 • 50.0@60.0' lOO.CH/Slatos* 

. <NPT> 

<NPT_Row>0.5«;/NPT_Row> 

<NPT Row>0.6-VNPT Row* 

<>NPT> 

<Ezpr*»ion Patldionod^tal**' S«nu(ation='U«o'>Unifonn<0.100><&pr«s«on> 

<Ob«»ivaiion«/> 

<N&de> 

<Nod# id=’M4_1_t_2_1_1_1_1_1_5* Nani#="1_R*lovan<#" Typo=Xontinuou* IntofVil** 

<Paronrt^*rM4eOM380M40</Par*ni,Nod«> 

<Stal*s*0.0-50.0@60.0- 100.0</Slatos> 

• <NPT> 

<NPT Row>0.091 .0^13^1 .0^1 0^1 .0^1 .090.7388421^1.091 091 . 091 391 39 I .O 91 091 .090.8842105491. 091 .090.7388421 9t. 

<NPT'Ron*0090 090390.090 O 9 O.O 9 O.O 9 O.O 9 O 263157990.0^ 0^ 090390390.090090.090.3157894690.0^.09 0 2631579 90. 
<yNPT> 

<Expr*ssion PaftitH>n*d='fal*a* Siniula)ien=*fal*a'>Arithinalic((M46*M364M40V3)-£xprassion:- 
<Obs*rvalions/> 

<Moda* 

• <FJod# Ninia=*23_W»o_Should_Act' Typa='Continuoua Intarval'* 

<Pa(»nl Noda > 

<Stalaa>0.0-50.0e60.0- 100.0</Slatt*> 

• <NPT> 

<NPT_Row>0,5</NPT_Rew> 

<NPT Row»0.5«/NPT Row* 

<44PT> 

<Exprassion Par1dionad=’falM’ SanLjia(ion=lalM'*Unifonn{0.100)«^pr*s9on^ 

«Oba*rvaiions/> 

<44ftda> 

■d4odt id=*M4_1_1_2„1_1_1_1^1_7' Nama=’30_Explain.WhBt* Typ*=’ConlintMus Interval' - 
<Pai*n1 Nada > 

<Sl8la$>0.0-50.0950.0- I00.l></Si«88> 

<NPT> 

<NPT_R«w*0.5</NPT_R«w* 

<NPT R«w*0.5</NPT Row* 

<iNPT> 

<Expr*ssion Paddionad-talaa’ Smutalion**falM*>Unifonn<0.100)<£xpros9on> 

«Obsaivaiions/> 

<^od** 

<f4od* *d=‘‘lllU_1_1_2_1_1_1_1_1_8* Nafln*=*32_L*ad_Tima_Providod‘ Typa*‘ConiinuoiM Intarval*:- 
<Pai*nl_Nftda > 

<Stal*«>0.0-S0.0950.0' 100.0</^at*s* 

• <NPT> 

<NPT_Row*0.5</NPT_R«w* 

<NPT Row*0.5</NPT Row* 

<iNPT> 

<Expr*»ion Parldionod^Tal*** Sanu<ation='Tal*o'*Unironn(0.100><€xpr««sion* 
cObsarvationi/* 

■«^od#* 
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• <No<lt Nam#=*24_Tim»_Window_To_A«f Typt^'Continueu* InMrvai'- 

cpaivni N»4i > 

cSt«l«s>0.0*50.0Q60.0- 100.0</St«ttt> 

• <NPT> 

<NPT Row>0.5«yNPT Rew> 

<NPT Row>0.5</NPT Row> 

<4JPT> 

<Expr*s*ion Par14ion«d=*falM* Sanul«(ion=7d»«*>Unifonn(0.100)<^prasiion> 

<Obstrv«lion«/> 

<AJod*> 

• id==*M27’ N«m«=*H««ring par Opt Out' Typ«-Xontinueu« Intarval’' 

<Piranl_NwJa>M4_1_2'yP»rart_Noda> 

<Stalas>>{nAnily * 0.0^ 0 - 10 0^10.0 • lnflniy</Staias> 

• «NPT> 

<NPT Row>0.0@0.0<4<IPT Rovr> 

<NPT>ow>0.0@0.0<4'IPT2Row> 

<NPT Row>1-0@1-0<4'IPT R«w> 

<iNPT> 

<Ezpra»ion Partitionad^trua* Simu<aticn=‘tnia'>AnthmalK(M4^1_2)M4_1_2@#<€*pfa»»ion> 

<Ob$arvations/> 

<Mada> 

• -44ode »d=*M24' Namas'Opt Out par Uncoordinatad' Typa^'Continuaus Intarval'^’^ 

<Paranl_NwJt>M4_1_1_2<^afan«_Noda» 

<Stala$>4rrfSnily • O.O^.O • IO.OQIo.O • lnlMty</$tatas> 

- <NPT> 

<NPT Row>0.0@0.0-:4>IPT Rew> 

<NPT RDw>0.078d4736S^.008771»3</NPT Row> 

<NPT Row>0.92105263@0.99122804<WPT Row> 

</NPT> 

<Exprassion Partitionad-lnia* Smul8lion-trua*>M4_1,1_2(Sf(inifonn(2.5,97.S}Triangla<2.S,97.5,70)<^prassion;‘ 

<Obaarvaiions/> 

•<^oda> 

• <fJoda 14=11129' Nama='Opt Out par Spam* Typa=*ContirHioua Intarval' - 

<Paranl_Noda:-M4_2-:/Paranl_N^> 

<Slatas>-{nfinity • 0.0@0.0 • 10.0^10.0 • lnfiraty</9alas> 

- <NPT> 

<NPT Row>0.0@0.0<4<JPT Row> 

-:NPT~Row>0.07894736S|gib.008771»3</NPT Row> 

<NPT Row>0.92l05263^.99122804-:^T Rov>r> 

<INP1> 

<Expra«$ion Panitienadsfru*' Simulat)ons'trva*>M4_2^IUnifpnn{2.&.97.$)Tnangta<2.6.97.$,70)<€xpras$ion> 

':Obaaivaiidna/> 

<^eda> 

• <Noda »d=*ll(t39' Nama=*Opt Out par Radundancy* Type=‘Canlinuoua IntarvaJ' - 

<Paran1_l4oda>M4_1_ 1 </Parart_Noda> 

<Stalas>-tnAnily • 0.0^ 0 • 10 oiglO.O • lnfinity</Statas:‘ 

- <NPT> 

<NPT Row>0-0®0-0<^PT Row> 

<NPT~Row>0 .07894736500.021531101 <^PT Rew> 

<NPT Rew>0.92l0S263O0.9784689-^T R0w> 

<AJPT> 

<Exprassien Paf1riM>nad=*tnja' Sanuia(ion=*tnM*>M4_1_1OfUndorm(2.5.97.5)Triangfa(2.S.97.5.30)«/Exprassier>> 

<Obsatv«iont/> 

<Meda> 

• <r4oda id=1431* Nama='3aliaving par Explain* Typa='ContirMieua lrMarvar> 

«Paran1_Noda>M4_1_l_2_1_1_1_t_1_7<Paranl_M»da» 

<Statas>-lnftnily > 0.0^ 0 • fo.^lO.O • lnKnily</Slalts> 

. «NPT> 

<NPT Row>0.0O0 0<4^PT Raw> 

«NPT’Row>0.00877193@O*00e77193«4«>T Row> 

<NPT Row>0.99l22804OO.99122804<^T Row> 

•:iNPT> 

<Expratsion p8t1ilionad=1rua'Simulalion=*lrua*>M4_1_1.2.1.1_1_1.1.70*Tnangla<2.5.97 5,70)Tnangka(2.5.97.5.70)</Exprass»on> 
cObsarvaiions/> 

<^Pda> 

- <Noda id=*M32* Nama^’Act par Action’ Typa-'CoRlinuaua IntarvaT^- 
<Paranl N»dt>M4 1 1 2 1 1 1J 1<.iParant Noda> 

<Slala«>-lnfinfly • O.O^.O • 10.^10.0 • lnfiniy</Stala$> 

• <NPT> 

<NPT Row>0-000 O^AJPT Row> 

<NPtIrow>0.00877193O0~0078424S4-:4^PT_Row> 

<NPT Row>0.99122804OO.9921575<NPT Row> 

<iNPT> 

<Exprassion Parlitionad='trua*Simulalwn=*trua*>M4_1,1^2.1_1_1_l_1^Triangla(2.5,97.5.70)Triangla(2.S.97.S,78.75><>Exprassion> 
<Obaarvaiions/> 

■^oda> 

• <^4oda 14=1433* Nama=*Act par Tima Window* Typa=‘Continuoua Intarval*^ 

<Paranl_Nodt>M4_1_1_2_1_1_1_1_1_9<Paranl_Moda> 

<Stalas>-lrrfinity • 0.0^ 0 • fO.OOIO.O • lnfinily:/Stalas> 

. «NPT> 

<NPT_Row»0.0O0.0>:jNPT_Row> 

<NPT Row>0.0Q877193O0.007842454</NPT Row> 
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<t«>T Row>O.»91220O4^»921S75tMPT Row> 

</NPT> 

<Expr*ssion ParMion«d=‘tru«* SmuMien=tnM*>M 4 _ 1 _ 1 _ 2 _ 1 _ 1 _ 1 _ 1 _ 1 _ 9 ( 8 #rh«n 9 l *<2 S.97.S.70)Tritngl*(2.S.97.S.78.7S)-vExpr*ssion> 
<Obstivilions/» 

• *f4od* id=^34* N«me=‘ 6 «li*vin 9 p«r Rcdundanc/ Typ*=*C«ntinuou« Inttrvar:* 

<P«r»nl_Nod#>M4_1_ 1 <P«rart_N«d*> 

<Sl«l«s>-lnRnity • 0.0^ 0 • 10 OQIO.O • lnfinity</St«lts> 

• <NPT> 

<NPT Row>0.0^ 0-:4^PT Row> 

<NPT Row>0.01246S374^.00877193<44PT Row> 

<NPT Row>0.98753464OO.991228(M<MPT Row> 

<iNPT> 

<Ezpr*ssion Parlilionad-tni** S■nulalk>o-*tnla*:■M4.1.t^l^Triangla(2.S.97.S,S0}Tria^gta(2-S,97.5.70)<£zprassian> 

<Obsarvaiions/> 

<^oda> 

• <h4oda id='M39'' Nama=’Act par Laad Tima* Typa='CeiitirHi(HiS intarval'^ 

-^Parent t^da>M4 112 11111 9<vPMnl r4»de> 

<Sla1as>'{nfinity • O.O^.O • fo.OeiO.O - lnfiniy</§talas> 

. «NPT> 

-:NPT Row>0.0@0.0-:4>IPT Rew> 

<NPT Row>0.00977193e0.007007183<JNPT Row> 

<NPT Raw>0.99122804^ 9929928<MPT Row> 

</NPT> 

<ExpraMion Pat14ionad=trua'Simulalion=trwa*>M4_l_1_2_1_1_1_1_1_8@frriangla<2-5.97.5,70)Triangla(2.5.97.5.87.5)</E«pras»ion> 
<Obsaivaiion&/> 

<Mada> 

• <t4oda id^’MSd' Nama^'Act par Ralavuica' Typa^Xentinuous lntarvar> 

<Parani_Nada>M4_1_1_2_i.l_1.1.1_6<Parani_hted#> 
cStalas>-lnftnity • 0.0^ 0 • 10.0^10.0 • lnlMty</Stalas:' 

. <NPT> 

<NPT Row>0.0@0.0<4<1PT Row> 

-:NPT~Row>0.06965944@0~007007183</NPT Row> 

<NPT Ravr:-0.930340S@0.9929928cAJPT Row> 

<4JPT> 

<EzprassionPaf1ibw)ad-'tnje*SimuMions‘tnia*'M4_1.1_2.1_1.1_1_1_5@VTriangla(2.5.87.5.12.S)TfMngla(2.5,97.S.B7.S]<iExprassion> 

<Obaarvations/> 

<Mada> 

• ■:Neda id=1437* Nama-*Act par Who Should' Typa^Xontinuoua Intarvar - 

<Para^l_Noda>M4_1_1_2_1J_1J.1_8«vPa^a^^_^♦oda> 

<StMas>'lnfiniiy • O.O^.O • 10.0^10.0 • lnjMy</Stata$> 

<NPT> 

<NPT_Row>0.0@0.0<4>1PT_Row> 

<NPT Row>0.00877193®0.007007183<:/NPT Row> 

<NPT'Rovr:>0.99122804@0 9929928-:if^PT Rew> 

<iNPT> 

<Exprassion Pai14ionad*‘trua'Sim(4alions*trua*>U4_1_1_2_1_1_1_1_l_6^7Tnangla(2.S.97.S.70)Triangla(2.S.97.S.87.S)<^prassion> 
<Obsaivalions/' 

<4«loda> 

• *Meda id=*M38* Nama=‘Ralavanca par Who* Typa-Xentinuoua tntarval'- 

<P8rant_Noda>M4_1_l_2_1_l_1_1_1_6</Paran1_M»da> 
cSlalasMnfinfly • 0~0^0~ ib.OQ10.0~ lnfin«y</Stalas> 

- <NPT> 

tNPT Row>0.(l®0.0<iWPT Rv»f> 

<NPT Row«0.007007183e<>.0070071«3<^PT Row> 

<NPT Rov>r>0 9929928®0 »929928<4^T Row> 

</NPT> 

«Expra»»ionPartil»onad=*lnja*Simola1ion=‘lnia*>W4_1^1_2_1_1_l_t_1_6®(ITriangla(2.5.97,5.67-5)Tiian9la(2-5.97.5.67.5)<yExpra*sion> 

<Obsarvtfiona/> 

<;^oda» 

• <Noda id='M39’ Nama=’ 6 aliaving par FinaT Typa^Xontinuoua lnlarvai’> 

<Para^l_N^>da>M4„1_l_2.1.1_1J.1.3<Paranl.^♦oda> 

<Stalas>-lnftnily - 0.0^ 0 - fo.O^IO.O • lnAnity</Stalas> 

- «NPT> 

<NPT Row>0.0d0.0<AlPT Rour> 

>^NPT Rew:‘0.02IS31tOl(gib.0O8771»3-:MPT Row> 

<NPT_Row>0.9784889@0.99122804<tJPT_Row> 

</NPT> 

<Ezpra»ion Pai14ionad=*tnia'Simulalion="frua'>M4_1^1_2_l_1_1_1_1_3^Tfiangla(2.5.97.5.30)Tfi8ngla(2.5.97.5.70)</Exprassioo> 
<Obaaivalions/> 

■^oda> 

• <Nodt id^lddO' Nama='Ralavanca par Explanation' Typa^XoMinuoua lntarvar> 

<Paf anl_Noda>M4_1_ 1_2_1_1_1_1_1_7 <Paranl_Moda> 

<Stalas>-{nfinity • 0.0^ 0 • fo.O^IO.O • lnfiniy-:/Stalas> 

- <NPT> 

<NPT Row>0.0@0 0«AIPT Row> 

<NPtIrow> 0 .0087719 3@o"008771 93<4^T_Row> 

<NPT Row>0.93122804@0.g9122804<4«>T Row> 

<iNPT> 

<Ezpra»ion Pai14ionad=*tnia'Simulalion="troa'>M4_1_1_2_l_1_1_1_1_7^Tfiangla(2.5.97.5,70)Tfi8ngla(2.5.97.5.70)</Exprassioo> 
>:Obaaivations/> 

■«^oda> 
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<No<lt Namt^'ilndwtUnd pm Uneeordin«t*d* Typ«=Xon(iniMU* inMrval' ■ 

<P«r*n1__Nod*>M4_1_1_2<^»r»nt_Nod»> 

<Sl«l«s>-lnfinily • 0*0^0 - 10.l}®lo.0 • lnAnity«/St«lts> 

• <NPT> 

<NPT Row>0.0^.0<4^PT 

<NPT Row>0.(l07007163^.021531101<MPT Row> 

<NPT Row>0.9929928e0.9784$89</NPT Rqw> 

<tHPT> 

<Expr*ssion Paftilioncd='liiM* Simul4tion=*tnl•‘^*M4_1_1_2®l^^nanglt(2.5,97.8,67.8>Tn•ng(•(2.8.97.S.30)<JExpr•sswn> 
<Ob$*rv«lion«/> 

<^ed*> 

• <t4od* id-'IMZ" Namt=*6eli«ving par Local* Typ«='Centinuous lntarval‘> 

<Par*nl_Nod»>M4_1^ 1_2':4>af»rt_Noda> 

<Stal««>'{nRnny • O.O^.O • 10.0®lo.0 • lnfinity</Stalts> 

• <NPT> 

<NPT Row>0.0^.0<4<IPT Row> 

<NPT~Row>0.0t246$374^.0O877193<MPT Row> 

<NPT Row>0.987S3464(gO.991228(M-:/NPT Row> 

<^PT> 

<ExprtMion Patldionad^'tnio* Siniulationstnio'>M4_1.1_2l99Triangl»(2.S.97.8.$0)Tn«ngl«(2.$.97.$.70)<;^pras»on> 

<Obs*ivaiioni/> 

<^oda> 

■>t4od» id-Uddd' Name^'Conlinnalion par Local* Typa-'ContirMioua Iniarval*^- 
<Pafani_Noda>M4_1_1_2'--/Pafant_Noda> 
cStatas>-lrdSnity • 0.0^ 0 • lO.O^lO.O • lnlMly</Stalas:> 

• <NPT> 

<NPT_Row>0.0@0.0-:4^PT_Row> 

<NPT Row>0.078947368^.07894736$^PT Row> 

<NPT'Rew>0.92l0$263^.9210$2$3<MPT R*^ 

<iNPT> 

<Ezpra»ion Paddionad^’trua* Siniulations*trua‘>M4^1_1_2@INJnifdrrn(2.$^7.5)Unfonn(2.8,97.5><Cxprassion> 

<Obsarvations/> 

'•^oda> 

-«Nodc td^'MId' Nama^^'Baliaving par Cenfirmatien* Typa^'Continuoua lritarvar> 

<Paftnl_Nodt>M0.1_1_2<^afart_Nod#> 

<Stala$>«lrriSnily • O.O^.O • lO.O^lO.O • lnlMy</$talas> 

• <NPT> 

<NPT Row>0.0@0.0-:4«1PT Rew> 

■:NPT_Row>0.00877193@0.00877193-:;NPT_Row> 

<NPT Row>0.99122804@0.99122804<4«>T Row> 

<WPT> 

'^Expressian Paditifrned^'true* Siniulation=*trua*>M0_1^1_2^l^riaogla(2.5,97.S.70)Tnangle(2.S,97.&,70)-^-^presawn> 

<Oba»rv8iions/> 

<^odt> 

• <^4oda >d=*f443' Nania='Baliaving par Why Typa==Xoiitinueuf Intarval'^ 

<Paran1_l4oda>M4_1_l_2_1_l_1_t_1_4<Paranl_Moda> 

<SMcs>>lnfirrity • O.O^.O • fo.^lO.O • lnAniy</Stalas> 

- <NPT> 

<NPT Row>0.0^.0<^PT Row> 

<NPT>evr>0.00877193QO'00877193-:4«>T Rew> 

<NPT Rew>0.99l22804Q0.991228(M<MPT Row> 

<#JPT> 

<Exprassion Pad4ionad=‘trua' Siniulalien='liiia*>M4_1_1_2_1_1_l_1_1_4^Tnangla(2 S.97$.70)Triangla(2.5.97.S.70)</Exprtss>on» 
<Ob$arvalions/> 

<^oda» 

*t4oda »d=*M4<* Namt=1)alavarKa par Why Act* Typa='Continueua Intarval’ ■ 

<Paran1 Noda>M4 112 11111 4<PBrtrd l<loda> 

<Slalas>4nfinity • 0.0^ 0 - Td.^IO O - lnfMty</§talas> 

• <NPT> 

«NPT Row>0.0@0 0</1^PT Row> 

<NPT Row>0.02«666667eO.0<jNPT Row> 

<NPT Row>0.97333336®1.0<jNPT Row> 

<iNPT> 

<Expra»sion PadilM>nad='trga*Smulation=*lnia*^M4_1_1_2.1_1_1_1_1_4(8ifrriangla<2.5.70.97.5)TNorni^78.2.0.100-0></Expfas*ion> 
<Ob$atvaiion«/> 

■<Mode> 

• -ttoda id^lM?' Nama='6aliavirig par Spaiiitg* Type=‘Cominuous lniarvaJ'> 

<Paranl_Noda>M4_1_1_2_1«VParart_Ncda> 

<Stalas>-{nfinity • 0.0^ 0 • 10.0^10.0 • lnlMly:/Stalas> 

• <NPT> 

<NPT_Row>0.0@0 0-:AJPT_Row> 

<NPT Row>0.01246S374^.00877193</NPT Row> 

<NPT~Row>0.987S3464@O.991228(M-:jNPT Rew> 

<AIPT> 

<Expra»ion Paddionad^trua* Sanulation='lrua*»M4_1_1_2_1@*n’rtongla(2.5.97.5.5O)Triangla(2.5.97.5.70)</Exprasston» 
<Obsaivationi/> 

■^oda> 

<Noda id='M49' Name=''Baliaving par Awaxanast* Type=*Continifeua Iniarvai* • 
cParant Noda>M4 112 1 loParanl Noda> 

<Slala«>-{nfinity • O.O^.O • fooeiOO • lnfiniy</5tala$> 

• <NPT> 

<NPT Row>0.0@0.0<4<IPT Row> 
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<NPT Rew>0.0l5789473l8O.0O877m<;NPT Row> 

<NPT Row>0.984210SS(8O.991228(M<^T Row> 

<^PT> 

<Ex|>rtssien ParlAion«d=1riM* Smiul«tion='triM‘>M4_1_1_2_1_1|gir‘har^)to{2.S.97.5.40)Tn«r>9tt(2.S^7.S.70)<^pr«st*on> 
<Obs*rvaiions/> 

<Med*> 

• <NQd« id=*M49* N«m«='Bdlidving pw Rtivvanc*' Typ«=XorMinuou» lnt<rv«r> 

<P«r*rt_Nod*>M4_1_1_2_1J_1*yPifwl_Nod*> 

<St0l«$>-lnftnily > 0.0^ 0 • fo.OQIO.O ■ lnfiniy</Stalts> 

• <NPT> 

<NPT Row>0.0®0.0«^4^PT Rtno 

-NPT Row>0.0557275$3e0,008771«</NPT Row> 

<NPT Row>0.94427246^.991229(M-:iNPT Row> 

<^PT> 

<Expr*ssion Par1ilion«d-tni«* SimuMton-*tnM*:-M4.1.1_2.1.1.1®#Tnangla<2.5.SO,97.S)Triangl»(2.S.97.S.70)</Ex|3r«ssM>n> 
cOb»trvaiion«/> 

<Mdd4> 

- -d^ede id^fd&O' Name^'Baliaving per Wasted AJcfts* Type°XeiMinueu» Interver- 
<Par»nl Npde>M60</Par«r4 Node> 

<Sl«t«s>'lnnnity • 0.0@0.0 • 10.0^10.0 - lnfiniy</5t«tt«> 

• <NPT> 

<NPT_Row>0.0@0.0-:4>IPT_Row> 

<NPT Row>0.0®0.0<^PT Row> 

•.-NPT’Row>1 .0@1.0<^PT"Row> 

<iNPT> 

^Expression Partilioned-*tru«* Simulation-*tnia*>M60^in’Nennal(77.3.0,100.0)TNerniaK87,9^,100.0}</Exprassion> 

<Obsaivalions/> 

•^eda> 

• 4<ieda IdBlISI’ Nam»='ealiaving par Sourct' Typas'CentinuPU* Intarvai*- 

cParam r4oda>M4 112 11111 1<Parani ^«oda> 

<Statas>'{nfinity- O.O^.O- fo.OeiO.o’- lnfiniy</§talts> 

. «NPT> 

<NPT Rew>0.0@0.0@0.0-^PT Row> 

<NPT Rew>0.00877193^.00677183e0 00877193<MPT Row> 

<NPT Row>0.99122804@0.99122804^.99122804«iNPT Row> 

</NPT> 

<ExpressionPai1ihoned=triia'Simulation=1ru#->M4J.1.2J.tJ.1.tJ@irTriangla(2.5.97.5.70)TfTang1a(2,5.97.5.70)TriangM2-5.97.5.70) 

•;£xprassion> 

<Obsaivaliens/> 

<^ode> 

• "tloda »d^M52‘ Narna^’Beliaving par Mora Info* Typas'Centinuoui Intarval'^ 

'Parem N»de:-M4 1 I 2 1 1 1_1 1 2<vParenl Node> 

<Stalas>-{nfinity • O.O^.O • fo.^lO.O • lnfinity</§talas> 

. «NPT> 

<NPT Row>0-0@0 0-:4'JPT Row> 

<NPtIrow>0 .0087719 3eo"008771 «':^T_Row> 

<NPT Row>0.99122804|9O.991228(M<^T Rew> 

<WPT> 

<Exprassien Par14ionad=‘trua‘Siniulaltens*inio'>M4^1_1.2.1.1_1_1.1_2^Tnangla<2.5.97.$.70)Trianglo(2.S.97.S.70)</Exprassion> 
<Obsarvaiions/> 

<Meda> 

• <Noda •d=1tlS3'' Namo=‘Spani par Alert Type' Typa=*Continueua bilorval’:- 

<Param_Noda>M4</Pai«nl.Noda^ 

<Stalas>-1nlWtily • O.OQO 0 • 10.0^10.0 • lnfMly</Slatas> 

• <NPT> 

<NPT Row>0.0®0.0Q0.0<>TgpT Row> 

<NPT Row>0.021531101Q0.02153ll01^.02153l101<JNPT Row> 

<NPT Row>0.9784689e0.9794689^.9784689<;NPT Row> 

<AIPT> 

<Exprassion Par14k>nad='truo'' Senulation=%iit'>M4@fTnangla(2.S.97.5.30)Triangla<2.S,97.5.30)Tri8nglt(2-5.97.S.30)</Exprassion> 
<^Obsarv8iions/> 

<^oda> 

- <l4oda id='MS4* Name=’Spam per RalavarKa' Type='Centinuous interval' - 
<P8ran1 Noda>M4 112 11111 5<Paranl Moda> 

^StalasMnfinjiy. O.O^.O - TO-O^IO-O - lnlMy</$taia$^ 

<NPT> 

<NPT_Row>0.0@0.0<4^PT_Row> 

<NPT~Row>0.00877193@<r08232687«tiPT Row> 

<NPT~Row>0.99122804@O.93767315<)NPT~Rew> 

<iNPT> 

-cExprassion Panilioned-*trua*Sinnulalion^'tnia*>M4_1_1,2_1_1_1_1_1_S^liTnangla(2.5.97.S.70)Tnangla(2.5.97.S,12.S)</Exprassion> 
<Obs»rvaiions/> 

'^oda> 

• <f4oda id-IISS’ Name='Undaratanding par Language* Type=*Continuoua interval* - 

<Parenl_Noda>M4_1_1_1_1</Pafart_Noda> 

<Stalas>-{nfinity • 0.0^ 0 • 10.0^10.0 • inlinity:/Statas> 

• <NPT> 

<NPT Row>0.0@0.0<^PT Row> 

<NPT>ow>0.0t246S374(^.007007163<:/T>IPT_Row> 

<NPT Rovr>0.98753464eO.9929928</NPT Row> 

<AIPT> 

<Exprts«ion Partitioned^true' Simulation=*mi#‘^M4_1^1_1_1@*n’riangla(2.5.97.5.50jTt1angla(2.5.97,5.87.5)<€xpression> 
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<Obs*rvalions/> 

<JN96*> 

■ <Ned* id=145S* Nam«='‘S|>«ni p«r AwaranM*' Typ«=Xontinuou« lrN«rval‘ ■ 

<Par»n1_Nod*>M4_1_1_2_1_l<^ar#nl_N«d#» 

<'StM«s>-lnf)nity • O.O^.O • fo.O^IO.O • lnfini(y</St«tas» 

• <NPT> 

<NPT Row>0.0^.0<^PT Rwf> 

<NPT Row>0.007007163^.021S31101<NPT Row> 

<NPJ Row>O.9929g28eO.9704$89</NPT Rew> 

</NPT> 

<Expr»»sion Partitionad-'tru*'' Smul«tion^lriit*>P44_1_1_2_1_1|gfTriingto<2.S.97.5^7.$}Trwigla{2.$.97.$.30)</Expr*s«ion> 
<Obi4rv«iont/> 

<AJod*> 

• <l4»d» id»'MS7* Nam«=*S|>«m par Ralavanca* Typa:*Continuous Interval' ■ 

<Paranl N»da>M4 112 11 1<VParfrd N«da> 

<State«>-lrrfinny • O.O^.O • fo.'^IO.O • lnriniy</Stalts> 

. <NPT> 

<tslPT Rew>0.0@0.0<:4^PT Rew> 

<NPT_Rovr>0.012485374eC.0W32«7<AIPT_Row> 

<NPT Rovr>0.98753464@O.9376731$<yNPT Revr> 

<>NPT> 

<ExprtMion Patlilionadsinia* Simulalioo="tr«#*>Mi_i_1_2_1_1_1(8ll^fMn9l«<2-5.97 S.$0)Trian9la(2.S.97.5.12-5)<€*prts*Kin> 
<Ob$arvaiions/> 

<Alod4> 

• <f4oda id-‘MSS' Natna='Spam per Spelling' Type^’Centinueus Interval' 

•=Pafenl_Nade>M4_1_l_2_1<VPafert_Node> 
cStates:-tn6nity • 0.0^ 0 • 10.0^10.0 • lnllnily</Stales> 

. <NPT> 

<NPT Rev/>O.OdO.O<AlPT Row> 

■:NPT Rew>0.07994736S|gib.01246S374<NPT Ro>v> 

<NPT_Rovr>0.92105263@O.987534W<^«»T_Rovr> 

<yNPT> 

^Expression Partitiontd-true* Simulalion=*true''>M4_1.1_2_1<SiUnif6rm(2.5.97.5)Triangle(2.S.97.S.50)<yExpress<on> 
«Obseivationi/> 

<Alede> 

• -<No(fe id- “MM' Nama-'Spam per Wasted Alerts' Typas'Continuous Interval'^ 

<Parenl_Node>M60</Parer4_Node> 

•:States>-trAnity • O.OigO 0 • 10.0^10.0 • lnllnily</States> 

- <NPT> 

<NPT Row>0.0@0.0<AIPT Row> 

<NPT'Rew>0.9999997^.0<^T Row> 

-^NPT Rew>2.886S164E-7@1.0<AJPT Rom:- 
<^PT> 

<Expressien Paitilioned'*true* Sinnulalion:*tnie*:-M60^TNormal(S,1.0.100.0)TNermBl<89.1,CI,100.0)</Expressien> 
<Obseivations/> 

<AJode> 

• <Nede >d='M88* Name^^'SS.Degree.Wasted.Alarts* Type=‘Cominueus Interval':- 

<Paienl_Nodt > 

<Siale$>0.0-80.0Q60.0- 100.0</Slales> 

• <NPT> 

<NPT Row>0.5<jNPT Row> 

<NPT Row>0.6-'/NPT Row* 

•«AJPT> 

<Expression Par14ioned=falte' Sanuialion=lalse'>Unifonn(0.100)<£Kpression> 

<Ob$eivalions/> 

<Alode> 

• <N»de id=1461' Name='Heering Per Spam* Type=*Cc»nCiniM>us Interval’* 

<Parenl_Node*M4_2<^arent_Node> 

<Stales>-tnftnily - O.OQO 0 • 10.0Q10.0 - lnfiniy</Stales> 

- «NPT> 

<NPT Rew>0.0®0.OcAJPT Row* 

•:NPT Row>0.024064t7e0.00877193-:APT Row* 
cNPT Row>0.97S9358e0.99122804-vNPT Row* 

<AJPT> 

<Expr*tsion Padiiionad^'tnie' Smula(ions-tnie'*U4_2(g^riangla<2.S.87.$.30)Triangl4(2.$.97.$.70}</CxprestieA> 
^Obeeivations/* 

■«Alode> 

• <Node id='M82' Name^'Understanding per Spelling' Type^'Continuoue Intervar ■ 

<Par»nl_Node>M4_1_ 1_2_ I <Parert_Node* 

<Stales*-{nfinity • O.O^.O - 10.0^10.0 • infinily-:/States> 

• <NPT> 

<NPT Row*0.0@0.0<4<IPT Row* 

<NPT~Row*0.0a877193@O~007007183</NPT Row* 

<NPT_Rew*0.99l22804@O.9929928-vNPT_Row:- 

<yNPT> 

<Expres«ion Paititioned=triie' 5ifnulation=*tnie‘*M4_1_1_2_1®#Triengle(2.5.97.5.70jTfiangle<2.5.97,5.87.5)<€xpression> 
<Obseivatieni/> 

■«Alode* 

• -t^ode »d*'M87' Name=''Spam per Redur>dancy Type=‘ContintM>us Interval'* 

<Parenl_Node*M4_1_1 </Pereft_Node> 

<Staies*-tnf)nity • 0.0^ 0 • 10 OjglO.O • inlinily-:/States* 

• <NPT> 
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<NPT Rew>0.0®0 0'-MPT Rew> 

<NPT Row>0.00877193^.06232687<MPT Row> 

<NPT Row>0.99l22804eO.9376731S<;T«>T Row> 

<#JPT> 

<Exi>r*stion Par19ion«d=trut* SmulJtion=1riM‘>M4_1_1i9#Tri«ngl*{2.&.97.5.70)Tnansl«(2.5^7.5,12.5)<^prassion» 
<Obsttvalions/> 

<Nod« »d='M88' N*m«=*Spam p«r Fraquaney* Typ«=Xon(inueut lnMrvd'‘> 

•:Par»n1_N)>d»>M4_1 •:>Parart_l'*Qil*> 

•:St«l«s>>lnAnily • 0.0(SO 0 • 10.0^10.0 • In8nily</Stalts> 

• <NPT> 

-NPT Row>0.0®0.0<4^PT Row> 

<NPT Row>0.06232687^.00877m-:JNPT R0w> 

<NPT Row>0.93787315®O,991228W<^T Row> 

<^PT> 

<Expr»ssion PaditM>nad-tni*' Smul4rtions'tnw'>M4_1@irrriangl«(2.5.97.5.12.$)Tri«i^2.8,97^.70)</Expr»ssion> 
<Ob$»tvaiion«/> 

-«t4od« td-'M89' Nam«='Und«r*tan4 par Eiay FaRaw' Typa='Canlinuau» lntarvar=> 
<Pafanl_Nada>M4_1_1_1<^afan«_Nada» 
cStatas^-lrAnily • 0.0^ 0 • lO.O^lO.O • lrAnity</Statas> 

• <NPT> 

<NPT Row>0.0®0.0<4<JPT Raw> 

<NPT>ow>0.00877193(SO~0070071$3<MPT Row> 

<NPT Row:-0.99122804^.»929828-:t4PT Row> 

<INP1> 

cExprassion Parlitionad-trua* Siniulalion=*trua^-M4_1,1_1@fTriangla{2.5,97.S.70>Triaogla{2.5.97.S.67.5)<^xpraasMn> 
<Ob$arvaiions/> 

<Mada> 

•>t4o<la id-IRZO' Name='6aliaving par Fraquancy* Typa=‘CerMinuoua Imarval*^- 
<Paranl Nada>M4 1</Paranl Nada> 

<Statas>'lnnnity • 0*0(90.0 • 10.0®10.0 - lnfiniy</9alas> 

• <NPT> 

<NPT_Row>0.0@0.0-:4<JPT_Row> 

<NPT Row>0.007007183^.021531101</NPT Raw> 

<NPT'Raw>0.9929828®0.9784S89<.WPT Re^ 

<yNPT> 

cExprassion Partibanad-trua* S«nulation-'trua‘:-M4_1(^^riang(a<2.S,97.S.87.5)Triangla(2.S,97.5.30)</Expraition> 
<Obsaivalioni/> 

•^eda> 

^Noda id^iyi/r Nama=’Cambo 19' Typa=‘Centinueus liMarval'^ 

<Parenl_Node:-M38@M37--/Parefil_Noda:- 
<Stalas>-{nfinity • O.OlgO.O • 10.0@To.O • lnfiniy</^alas> 

<NPT/> 

<ExpraMion Paititionad=‘Maa* SanuiatM>n-*tnia‘>Anthmalic((M36-»M37)/2)<^pras«an> 

<Obsarvaiiens/> 

<^ada> 

<Noda id-1l(l72* Nama^'Combo 12* Typa-'Condnuoui lnttrvar> 

<Parant_Nada^M3&6M33</Parar4_Neda> 

<Stalas>-lnfinity • 0.0(90 0 • 10.0^10.0 • lnfinity</Slalas> 

<NPT/> 

<Exprassian Paftitionad=*falaa* SMnu(ation=*trua*>Arithmalic((M3S4M33y2)</Expratstan> 

<Obsatvalidnt/> 

<^ada> 

-^da id=1N73* Nama=‘Comba 13* Typa=Xen(inuaua liMarval*:* 

<Paranl_Noda>M32AM72<Parara_Noda> 

<Stalas>4nftnily > 0.0(90 0 • lO.O^To.O ■ lnfMty<(Sla(as> 

<NPT/» 

<Exprassien Partilionad=*falt#' Simuiatiert=*mia'>Aridimatic((M324M72)/2)</Exprassian> 

«ObsarvalionV» 

<^ada> 

'f'toda id=^74' Nama=’Camba 85' Typa='Centinueus IrNarval*^- 
<Paranl_Noda >M&8®M57- :/Para<4_Noda > 

<Slatas>'{nfinity • 0.0(90.0 • 10 OQ10.0 • ln(iraty</Stalas> 

■eNPT/> 

^^Exprasaion Pariitionad-'falaa' SanulatiQns'tnia*>Anthmalic((M89'*M57)/2)</Expras«an> 

<Obaarvaiions/> 

<^ada> 

<Noda id^WS' Nama=Xamba 84' Typa=Xantinuaua liKarval*^ 

<Paranl_Nada>MS6@M74-:/Paranl.Noda> 

'cStalas»-tnfinity • 0.0(90 0 • 10.0^10.0 • infiniy-:/Statas> 

<NPT/> 

<Exprassion Paftitionad=*falsa* Simulatnn-*trua*>Anthinalic((M56+M74)/2)</Expras3ion> 

<Obaarvaiioni/> 

<^ada> 

<Noda id*'M7l* Nama=Xamba 83" Typa=*Centinuaua liKarval*^ 

<Paranl Nada>M7S@M58</Paranl Noda> 

<Slata«>-(nfinity • 0.0(90.0 • 10.0@To.O • lnfiniy</Stala$> 

<NPT/> 

<Expra»ion Partitienad=‘Maa‘ Sanulation=t™a*>Anthmalic((M75-tM58)/2)<JExpfawion> 
cObsaivaiioni/> 

<^ada> 
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<No<lt id=^77* Nam«=‘Coinb<» 92* Typ»=*Centinueu« IntervAl* ■ 

<P«r*rd.Nodt>M98QM67</Partftf_Nod*> 

<:Sl«l«s>*4nfinily • 0.0(80 0 - 10 0^10.0 • lnMly</St«lts> 

<NPT/> 

<Exi>r*Mion P8rtition«d=*falM‘ SNnu<«tion=‘trut'>Anthm«lic{(M$64M67)/2)<>Expr*tston> 

<Obstrvalions/> 

<Nod* id=197t* N«m«=Xombo 19* Typ«=Xentinueu« lrN«rvil‘> 

• :Par»nt_N<>de>MS2®M51 </Parffd.Ned«> 

•:St«l«s>>lnAnily • 0.0(80 0 • 10 0^10.0 • In9nily</Slalts> 

<NPT/> 

<Exprtssion Paftition«d-*fals** Smulation-*tnit*'Arithtnelic((M52-»MS1)/2)</Exprtssion> 

<Obi*iv«iioni/> 

<^ed*> 

<f4od« id-*M79' Name=Xombo 99’ Type^Xendnueus IrNarval'"- 
<Par»nl_Nod«>M50(8M49<'Partfd_Nodt> 

<StM«s>-{nftnily • 0.0(80 0 • lO.O^To.O • lnfin(ty</5t«l«s> 

<NPT/> 

<Expr»»ion Paditionad-’M**' SMnulaticin='tnit’^Anthm«lic((lli1$0^49)72)^/ExprfMi9n> 

<Obs»rv«tions/> 

-44ode id=*M99' Nama=X9mbo 97* Typ**Xentinueus lntarval‘> 

<Par»nl_Nod«>M79QM48<>Pvtrd_Nodt> 

<Stal«$>4nlSnj|y • 0.0(80.0 • 10.0^10.0 • 

<NPT/> 

cExprtssion Paititionad^alaa* Simuiation=*trua*>Anthm«lic({M794M48]/2)*:/Expr*ssion> 

<Obs*ivalioni/> 

<^edt> 

"l^odt 19=1491* Nam«=Xdmbo 99' Typ«=Xentinueus intarval'^ 

• Parard Noda>M80@M47<P«rtAl Nod*> 

<Slal«s>-{nfinity • 0.0(30.0 • lO O^To O - lnriniy</^a(ts> 

<NPT/> 

cExprassion PaftiUon«d=*fala«* SmidatiM-*trua‘>Arithffl«lic((M90-»M47)/2)-:.€xpr*uion> 

«Obs*ivationi/> 

•^edt> 

-■ttoda id=1ill92* Nama^Xombo 15* Typt^Xontinuous Interval'^ 

<Par»nl_Nod«>M54(8M70</Partrd_Nodt> 
cStMas>-lrAnily • 0.0(80 0 • lO.O^To.O • lnlMy</Stalas> 

<NPT/> 

<Exprt»ion Partition«d=*Mf«' SMnu(ation='tiua*>Anthin«lic((M34-*M70}/2)<>Expras«i«n> 

<Ob«atvaiion«/> 

■<^ade> 

<Nodt 19=1193* Nam«=Xombo 19* Typ«=XerMinuoua Intarval*^ 

<Par»nl_Nod«>M42(8Ma</Par«fd,Nodt> 

<Stal«s>-tnfinity • 0.0(80 0 • 10.0@To.O • lnAnity</Stalas^ 

<NPT/> 

<Exprassion Paftition«d=*falaa* Smuia(ion=*trua*>ArithtR«lic({M42'tM44)/2)-:)Exprassion> 

<Obsaivalions/' 

<Modt> 

<Neda id=1l94‘ Nama=Xombo 17* Typa=XendnuMia Intarval*- 
<Parant_Noda>M93®M82</Partfil_Noda> 

<Slal«s>-lnfinny • 0.0(80 0 - 10.0®To.O • lnAn«y</Stalas> 

<NPT/> 

<Exprassion Paftition«d=*falaa' SNnulation=*trua'>Arithmtlic{(M934M82)/2)</Expr*ss(on> 

<Ob$aivalion«/» 

<A«loda> 

<Noda id=’MI5* Nam«=*Baliaving par Ratavanca* Typa=Xenlinuoua lnlarvar> 

<Paran1_Noda>M4_1_1_2_1J_1_1_1_5<Pafart_M»da> 

<Stalas>-4nftnity • 0.0^ 0 • fo.^lO.O ■ lnfM1y</Stalts> 

• «NPT> 

«WPT Row>0.0(80-0</NPT Row> 

•:NPT.Row>0.0a877193®O.00877m<tJPT_Row> 

•:NPT Row>0.99l22804@0.99122804<MPT Row> 

<A4PT> 

<Expratsion Patl4ionad='tnia' Simulai»n=*tnia'‘>M4_1.1_2.1_1_1_l_1_5^Tnangla(2-5.97 5.70)Trian9*a(2.5.97.5.70)</Expra$s»on> 

<Obaeivaiidns/> 

<^oda> 

<t4oda id=1(l99* Nama=Xombo 29* Type^Xentinueus liMarval*^- 
'cParanl_N»da>M4&@M8S-:/Para<d_Noda:- 
<Stalas>-tnfinity • 0.0(80 0 • 10.0^10.0 • inlinity-:/Statas> 

<NPT/> 

<Expra»ion Pa(tit(onad=*Nlaa* S«nulation='trua‘>Anthmatic((M45-»M85)/2)<)Expras»on> 

<Obaarvaiioni/> 

<^oda> 

<No<ta id=1187* Name=Xombo 21" Typa=Xontinu©ua ln<arval*=> 

<Paranl_N»da>M31@M86</Para<il.Noda> 

<Stalas»-tnfinity • 0.0(80 0 • 10.0^10.0 • inMly</Statas> 

<NPT/> 

<Exprt»ion Pa(1itionad=‘Maa‘ S«nulation='trua*>Anthmalic((M31-rM89y2)</Expra»t«n> 

<Obaarvaiions/> 

■^oda> 

<Noda id=148l* Nama=X«mbo 22* Typa=Xentinueua liNarval*^ 
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<Par*nt_Nod*>M39®M87</Part/M_Nod*> 

<S(al«s>-lnfinity • 0.0®0 0 - lO.O^To.O • lnfinity</St«lts> 

<NPT/> 

cEx|>rtssien Partitioned =*fal««* SMnula(ien=*trua*>Arithinalic{(M394M87y2)</Exprauion> 
<Obsarvations/> 

<^da> 

• <Noda id=1^8l’ Nama=X®rrtbo 11“ Typa='Cen6nueua lrNfrvar> 

<Parant_Noda>M84^M88</Parffd_Noda> 

<Stala$>-inAnily > 0.0(80 0 • lO O^To.O ■ lnfMty</Stalts> 

<NPT/> 

<Expra»sion Partrtionad°*M«e* Smula(ion2*trua*>Anthmalic((M84-»M88)/2)</Expr*ssion> 

<Ob«drvdiiont/> 

<^da> 

- <No(la id^mi' Name-’Combe 08“ Typa-'Cenliniieus IrNarval*^- 
<Par»nl_N»da>M$SOM62</Paranl_Nodt> 

<Slata«>-{nfinily • 0.0(80.0 • 10.0@To.O • ln(iniy</Stalts> 

<NPT/> 

'Expression Paititioned^alse* Simulalien-*true*>Anthmeiie<(M&S+M02V2)*vExpressiO(V 
<Obsarvaiions/> 

<jNode> 

• <Node id^HilOl* Name=’C«mbo 14’ Type^'Centinuous Interval':- 

<Par»nl_Node:-M41@M69-:Parent_Node> 

^SlatesMnRnity • 0.0(80.0 • 10.0^10.0 • lnfiraty</^a(ts> 

<NPT/> 

<Ezprts«ion Partition«d=*Mse' Sanulationstnie'>Arithinalic{(M4l4M69)/2)<>Exprtssion> 
<Obseivalions/> 

<^4ode> 

• -ttede id='W2* Name=Xombo 11" Type^Xentinueus Interval** 

<Parenl_Nfrda:>M789M89<^4renl_Ned#* 

cSteles:-(rAiity • 0.0(80 0 • lO.O^To.O • lnlMty</Steles:> 

<NPT/> 

<ExprtMion Partitioned^'Mee' Senulation=1nie*>Arithinetic((M76-HM89}/2)<>Expression> 
cObseivaiions/> 

<^itoda> 

• <t4ode id-*M93* Name-Xembo 81" Type-Xentinuous Interval':- 

vparem Npde>M76^$3<5>»ard Node> 

<Slates>'(nfinity • 0.0(80.0 • lO.O^To.O - lnfinty</^a(es> 

<NPT/> 

cExpression Partitioned ^alse' Simuiation=*true*>Arithme(ic((lil764MS3y2)<.€xpression> 
<Observaiions/> 

<^de:‘ 

--.^tisfc Objed> 

</Rrsk_Modat> 
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Appendix T Alert Originator Trust Model Formula Extract 
from AgenaRisk 


<?itml •rtcoding*TJTF-8“?> 

<Ri9fc_Mod*l Numb*r_0f_Ri8k_0bj*cls=*1 *> 

<Unks_btlw»»n_Risk_Ob^s^ 

• <Ri8k_0bf*ct Ngmb»r_0f_N*d«=74' ShQ(1_Otscnfrfion='N«w Risk Obisct' »d="N*w Risk Obj*ct_f'> 

<lnpuf_Nbd*s/> 

<OLJtpul NodssA- 

- <NcHi*id='M0‘Niim«=-vrEAUtili2ation'Typ»=-conarH>ouslnterv«l'> 

<Parsnl_Nodt>M1®M3@M2<)Parsnt_Ngds> 

<Stfll«S'»lnfinity • O.OigO 0 • 10.0^10.0 • inAnity*:/St«tss:- 

• <NPT> 

<NPT Row>1.0@1 0@1.0®1.0<^PT R9tf> 

<NPT~Row>1 .0^1 0^1.0^1.0^1.0^1 t.0<^PT''Rbw> 

-:NPT_Row>1.0@10@t.0^1.0^10^1 0@l.0@t.0<MPT_Row> 

</NPT> 

<Exprsssion P8rtilign*d='r*lt** SimulfltKin-*tni**>Anlhm«(ic(((M1/100)*(M2/10O)'(M3/100))*l00)<£xprsssion> 

<Obssrv«iions/> 

<Mpd«> 

- <Nede Name^’Appropriste' Type='CofWinifeus Intervar - 

<Parsnl_N»ds>M42_1^42@M50<;P*rs«4_Nodt> 

<St«lfs>0.0-SO.O@60.0'100.CH/Sl8tss> 

• <NPT> 

<NPT_R«m>I 8^1 0^1 OQl OQI 001.0^1.0^1 DQl OQI 0^1 0^1.1)^1 66< OQt 0^1 0Q1 0^1 OQl 0^1.061 A 

<NPT ReM>1 .0®1 0ei OQI Oei OQI 0^1 0^1 OQI O^t .(>@1 .OQt .0^1 0^1.0^1 0®' CQI OGI 0Q< <>61 OG' dQI OQI .061.001 A 

</NPT> 

«Expr«ssionPvti(ion«d='ralS«'Simulabon=‘r4lt«‘’Arithmstic((((1/M42)/((1/M42)-HiyM42 1}4<1M50)))‘M42)4-(((1/M42J)/((1/M42)-«- 
(1/M42_1)«(1MSO)))’M42_1)«(((iyMSO)/((1/M42)-Kl/M42J)«<1/M&0)) rMSO»</E>ipr*ssk»> 

<Obssiviiions/> 

<A4ods> 

• <'N»ds id=*M2* N«m«=’Aviil8bU‘ Typs='Continuout IntArval*'- 

<Par»nl Nodt>M786M79®M80</Parsnt Nod*> 

<St«l«s>0.0-$0.0@80.0- 100.0</S«tttt> 

. <NPT> 

<NPT_Rom>1 061 061861 86* 86* 86* 86106* 86* 86* 86* 86* 86* 86* 86* 86* 86* 86* 86* 86* 86* 86* 86* 86* 06* 8 

<NPT Rwi>1 06* 86* 86* 86* 86* 86* 86* 86* 86* 86* 86* 86* 86* 86* 86* 86* 86* 86* 86* 86* 86* 86* 86* 86* 86* 8 

<4»IPT> 

«Expr«ssionParttion«(^’rals«’SfTiulabcn-*f0lSC'»Arithmt(ic((«1/M78)/((17M78>+{1/M79)'*<1/M80}))*M76)t(((1/M79)/«1/M76H<1/M79)* 
(1/MeO)) rM79)4'(((1/M80)/((1All78><K1/M79)-Hl/MeCI}) rMeO))</Expr«ssion> 

<Obs*rv«iions/> 

<A^eds> 

- <Nodt id*'MJ’ Nams=‘Eff«c(iv*' Typs="Continuo«s Intsrval* • 

<Par»nl Nods>M716M$4^Partfll Nod4> 

'Stales>0.0-00.0680 0-100 0'-ySiMes> 

• <NPT> 

<NPT Row>1.06* 86* 06* 06* 86* 86* 86* 86* 8</NPT Row> 

■:NPT~Row>1.06* 06* 06* 06* 86* 86* 86* 86* 8</NPT~ Row> 

<>NPT> 

<Exprassion ParUionad^faiaa’ Sunuialion-'falM':-Arittvnalic(«(1/M64y((1MG4>«<1M71)))’M64>4<«iyM71)A(1/M04)4<1/M71)))'M71)) 
<Expression> 

<Qbssrvations/> 

<^oda> 

• <Moda id-'MO' Nama^'GMgrapNc Breadth' Typa^labeMad’ - 

<Parsn1_Neds > 

<St«l*s>78% aultids r«ns6S8 ouisids 2 m« 630 : outsid* ze4i*6l8% auttid* zana-^ Stiiat> 

• <NPT> 

<NPT_Row>0.25</NPT_Rw> 

<NPT Row>0.28</NPT Rw> 

•:NPT~Row>0.25<>NPT"R©w> 

<NPT ftow>0.25<^JPT Rw> 

<>NPT> 

<Exprassion Parlibonad^Taha' Siniulation-Talse'/> 

<Qbservations/> 

■<^ada> 

• ■:Nede id-'MO* Nams-'Rcsponsibility* Typs^'Labelled*^- 

<Pai»n1_Nads > 

<StMa$>SO% Canftdanca Ifs Ydur$670% Cenfidenca Ifs Yaurs680% Catiftdanca ir« Your«689% CanUanca ir$ Yours <^/S(aia$^ 

-:NPT:- 

<NPT_Row>0.25</NPT_Rqw> 

<NPT3Row>0.25</NPT~Raw> 

■:NPT_Raw>0.25<^PT_Raw* 

■:NPT Row>0.25<AJPT Raw* 

<iNPT> 

<Exprtssion Pa<titK>nad=Yalsa' Siniulation=Talse'/* 

<Obsaivations/> 

-^oda> 

- <Nodt id='M0’ Nama^Tima of Day" Typa=*Labtlled*> 

<Paran1_Nadt > 

<Slalat>lO:3ft 8m6830 pm67:88 affl-:Slatas> 

- <NPT> 

■:NPT_Row>0.3M33334-:/NPT_Raw> 

<NPT Row>0.33333334</I«>T Raw* 

<NPT>ow>0.33333334-:/NPTlRaw> 

<yNPT> 
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<Expr«ss*on PartMion*d=^*l*«* Smu(a(ion3*falM*/> 

<:Obs»ivaiions/> 

<Mod*> 

• id=1il11* N«ma='SyM*m Raadinns* Typ«=\aball*tf*> 

<Pai»nl_Nft4« > 

<StM«s>Un«vMla()<* 1 hrp«rw**lc®Un«vsilebla Ihr parmenthQUnavailabl* 1 hrparyMf</Statas> 

• <NPT» 

■=NPT Row>0.3333333a</?«>T Row> 

■:NPT Row>0,33333334</NPT Row> 

<NPT Rew>0,33333334</NPT Row> 

<^IPT> 

<Exprtssion Paftitionad-Talsa’ Siniiulation-Talsa*/> 

<Obaaivaiioni/> 

<^eda> 

• <tsloda id-'M13'' Nama=*SyMam Accaaiibilit/' Typa-'Laballad^- 

<Paitnl_N»dt > 

<Stalas>S*viral Dasijnaiad ind RamcU^Savanl Oasignalad^nmafy Ofka< States^ 

- <NPT> 

<NPT_Row>0.33333334</?«*T_Rw> 

<NPT Row>0.33333334</NPT Row> 

<NPT’Row>0.33333334</NPT”Row> 

<INPJ> 

<Exprassion PAiibanad-Talta’ Siniula(ion-*falM*A> 

<Obsaivaiiont/> 

■c4<lBda> 

• <Moda id-‘M14'' Nama-’Sysiam Rdiability' Typas'Laballatf^' 

<p8i»nl_N6da > 

<Stalas>'99.9 Statai > 

. <NPT> 

•:NPT Row>0.33333334</I^T Row> 

<NPT Row>0.33333334</T«*T Rbw> 

<NPT Row>0.33333334</?«*T Row> 

<>NPT> 

<Exprt»ion Pa<titi(>ntd3*falsa‘ SiniulationsYal«a‘/> 

<Ob$»rvaiions/> 

<Woba> 

- <Noda td'iaiC* Nam*="Sa«urit/' Typa=T,aballad*» 

<Par»nl_N6da > 

<Stal*t>N6 aflads in past yaai^ntueeaulil in patl v*ar^Suecasskil in patl yaar par usanQSuecasskI in past yaar par agancy-: Statar- 
. <NPT> 

<NPT Row>0.3$<jNPT Rdw> 

<NPT R0w>O.2S<4^PT Raw> 

<NPT_Row>0.25<)NPT_Rflw> 

<NPT Row>0.25</NPT Rw> 

<iNPT> 

<Expraasion Partitionad^falaa' Simulation-faha*/^ 

<Obsarvaiion«/> 

<Modt> 

- <Meda id=ll417* N8ma=*Magnituda of Effort' Typa 2 \aballad‘> 

<Pafanl_Nada > 

<Stalat>5 mintQIO mintQ20 mint^O mim': Slalat> 

• <NPT> 

<NPT Row>0.2S<MPT Rbw> 

■:NPT Row>0.25<4JPT Row> 

<NPT Row>0.25<4'IPT Row> 

<NPT Row>0.25<4'IPT Row> 

</NPT> 

«Expra»sion PartitK>nad='fal»a' Siniul8(ion=Tal«a*/> 

<Obsarvaliona/> 

<^oda» 

• <Noda id^fillB' Nama^'Croaa Syotom tntafiration* Typa=1ab«llod'> 

<Paranl_Nada > 

<Stalas>Noinla9ation0nla7alo4loolharsysta(nx :>Slalas ■ 

. «NPT> 

<NPT_Row>0.6</NPT_Row> 

'^NPT_Row^O.S<^PT_Raw> 

<INP1> 

<Exprassion Paftibonad-falaa* Simulaiion=*fala#*/> 

<Obaarva(ioni/> 

■^Bda> 

• <Noda id^'Mig* NatnasTanvIata’ Typa=*LabaMad*:- 

<Paranl_Nada > 

<Stalas>WEA lampfaias avaiablaigWEA lamplatas not availabla-^/Stalas- 

• <NPT> 

<NPT_Row>0.5<^PT_Row> 

<NPT Row>0.S<MPT Row* 

<iNPT> 

cExprassion Panibonadsfalaa* Siniulaiion=*falaa*/> 

<Obaarvationi/> 

<^oda> 

• -taoda id~‘M2g' Nama=*SI(illa Cornpeaancias' TyT>a=1.abaia<r> 

<Paranl_Nada > 


CMU/SEI-2013-SR-021 | 201 


<SM«t>40 hn traininsQIS hn traims^ fire training< SlMat> 

- <NPT> 

<NPT Row>0.33333334</r«>T R»w> 

<NPT Row>0.33333334</I«>T R»w> 

<NPT Row>0.33333334</I«>T Row> 

<4»IPT> 

<Exprts<ion P«rtition*d=T«l»4* Sinnula(ion=Talt«'/> 

<Obs*rviiions/> 

• •Mod* id=1l31* Namo^'Undarstifiding* Typa^XaMlod*'-* 

<Pai»nl_Nod* > 

<StMas>Thorough^odt<alaQMranal'VSi«*s> 

- <NPT> 

<NPT Row>0.33333334</?^T Row> 

<NPT Row>0.33333334</?^T Row» 

<NPT Row>0.33333334</?^T Row» 

</NPT> 

'Ex^ession Partitioned-Talae* Simulatien-*false*A> 

<Oba»rvaiions/> 

■«Mod*> 

• -Meda id=1l22* Nama^Pracliea* Typa^LabaMad*:- 

<Pai»nl_N»d* > 

<Stal*s>Tiw* par fiaak^Tfiic* par monli^TMica par y**r<Sta1*s> 

. <NPT> 

■:NPT Row>0.33333334</NPT Row - 
<NPT Row>0.33333334</NPT Row> 

<NPT Row>0.33333334</NPT Row> 

<MPT> 

<Expra«$ion Padihonadstal**’ Siniula(ienstal$»’/> 

«Obs*ivalions/> 

■«Med*> 

• -Mod* id=*M23* Nama='Timalino*s of Dtssarnnatien' Tyt>*='L*bollod'> 

<Par*nl_Noda > 

cStMa:* c 2mins@2-& mins@S-10 mins£10-30 rnm<ySlBl*s> 

• «NPT> 

<NPT Row>0.25</NPT Row> 

<NPT_Rew>0.25</NPT_Row> 

■:NPT_Row>0,25<MPT_Row* 

<NPT Row>0-25<MPT Row* 

<MPT> 

<Expra«$ion Padihonadstal**’ Simulatienstalf*’/* 

':Obaaivaiiena/> 

■«Mod*> 

• <Nfrd* id^M24'' Nama=‘Publi€ Awaranass* Typ*=XabaNa<r > 

<Pai*nl_Nodt > 

<Stalas>Pid)lic praviously informad of WEA^ubbc not praviously inforraad of WEA</Stalas> 

- <NPT> 

<NPT_Row>0-5<MPT_Row> 

<NPT“Row>0.5«/NPT"Row* 

<yNPT> 


<Exprassion PaftilK>nad=*fal*** Simulation-Talaa*/* 

<Obsaivalions/> 

<Moda* 

• •Moda id=*M2$' Nama=’Sy»t*fn Faadbaek* Typa=*Continuou* IrNarval*^* 

<Par*ni Noda*M&2QM53</Para<4 Nod*> 

<Stal*s>0.0-SO.OQ60.0- 100.CI</SlMa*> 

- <NPT> 

<NPT Row>1.0®1 Odt.OQI.OQI.Oei 0®1.0ei.O®1.0<M»T Row* 

<NPT Row>1.0®1 OeV0ei.O®1.0ei.O|SI.0ei.0®1.0</NPT Row* 

</NPT> 

<Expr*»sion Partritionad^fal**’ SiniiMion='tal*a’*Ariltm*tic((({1/MS2M(1MS2>a<1MS3)»’M52>a<({1/M53)A(1/M$2)^1/M&3»)'W$3)) 
•;£xprassion> 

<Obsaivaiioni/* 

<Mod*> 

• •Mod* id='M2$’ Nama^'Alart Fraquancy* Typa='Laballad'* 

■'-Paranl.Nade * 

<Statas>Sav*ral in past i-iaak^avaral in pad menfi^Savtral in pad yaar< Stains* 

• <NPT> 

<NPT Row*0.33333334-:/NPT Row* 

<NPT Row*0.33333334-:/l«>T Row* 

<NPT_Row>0.33333334</NPT_Row* 

<MPT> 

<Exprassion Padilionad^Talta* Sinnulaiion=*fal***/* 

<Obs*rvations/> 

<Mod*> 

• <Nod* id-'W37* Namo='Aftar Action Raviaw Data' Typa=Xab*lad‘* 

<Pai*nl_Noda > 

<Stal*s>UnlBVorabl*9N*utral^avcrabl*-: Statas 

• <NPT> 

<NPT Row>0.33333334</I«>T Row* 

<NPTlRow*0.33333334-:/b«»TlRow* 

<NPT Row>0.33333334</I«>T Row* 
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<WPT> 

<Expr*ssion Paftilion»d=*f«l»«‘ Simulation=Taf»«*/> 

<Obs»iviiions/» 

<Mod*> 

• <tJedd id=‘M29'' N*m«=Und»rmr>dabitit/ Typ«=*tabbllbtf'^ 

<Pai*nl_Nodt > 

<St«l«t>Only tM mt 9 t^u<iQm mu 90 chan^Mtam max 180 cbart^uttom max 270 chan<Sta1at> 

• ■'NPT> 

■:NPT Row>0.2$<>NPT R«w> 

<NPT“Row> 0 ,25<>MPT*Row> 

<NPT Row>0.25<>NPT Row> 

-NPT Row>0.25</NPT Row> 

<iNPT> 

<Exprassidn Partilionad''Talsa‘ Simulaiien='falta*/'-' 
cObaarvaiion«/> 

<^oda> 

• 4Jod0 id='M3(l’ Nama=*Public Faadback’ Typa='lab*llad'> 

■'-Paienl_N»de > 

<Stalts>Unbvofibla@Natftial@favorabl«-= Stall t- 
. «NPT> 

■:NPT Row>0.33333334</NPT Row> 

<NPT_Row>0.3M33334</NPT_Row> 

<NPT Row>0.33333334</I«>T Row> 

</NPT> 

<Ex(>rtMion Pa<1itH>ntd-falsa' SimulaiionsTal«a‘/> 

<Obsaivalioni/> 

<Mada> 

• <Nodt id-*M3r Nama-'Hiftorical Faadback* Typa='LabalM'> 

<Piitnt_Nada > 

<Staias>Nof»adbadc^a*dbackoftime]inasa@Faadbackofunlim»lina»«-:/SlaXts:- 
. <NPT> 

<NPT Row>0.33333334</I«>T Row> 

■:NPT~Row>0.33333334</NPT’Row> 

<NPT Row>0.33333334</NPT Row> 

<^PT> 

<Ezprtssion Partibonad-Talsa' Simulation-Talst'/> 

<Obs»rvations/> 

<Mada> 

• <Noda id-1132* Nama^'Raahinw Feedback' Type='Laballad‘> 

<Paitnl_Neda > 

<:StMa$»Nona(glAekRae’d^ekRa^dAeefpiad^ekRae'dAceap(adFbtwardad^ckRae’dAccapiadForwardadTran«min»d'^/Staias^ 

<NPT> 

<NPT_Row>0.2</NPT_Row> 

<NPT Row>0.2<A^PT Row> 

<NPT’Row> 0 .2 <iNPT’Row> 

<NPT_Row>0 .2 <iNPT_Row> 

<NPT Row>0.2</NPT Row> 

<WPT> 

<Ex|>r»ssien PaftriH>ned=1al*e* SimUationslalea*/^ 

<Observaliens/> 


• <N»de id=*M33* N8me=‘Maaaaga Accuracy Type=‘l.abaliad‘> 

<Pa(»ni_Nada > 

<StaXas>Pnorne trrorsQPnor miner arrera^PrtermejererTors</Slales> 

• <NPT> 

<NPT Rew>0,33333334</r^T Row» 

<NPT Row>0,33333334<>7#»T Row> 

«NPT Rew>0.33333334<yFPT Row> 

<AJPT> 

<Exprassion Partilionad=Talsa’ Simula(ion=Talst'/> 

«Obaervaiions/> 

Nede> 

- <Mode id=H34’ N8me='LaeBtien Accuracy* Type=*L^allad*> 

<Paranl_Neda > 

<:Staias>Pnor Wrong Araa@Prior Corrad and Soma Wrong Araas^iPner Only Podien of Corrad Araa</5taia$> 
'-NPT> 

<NPT_Row>0.33333334«:/?«»T_Row» 

<NPt'Row>0.33333334«:/?«»t'Row> 

<NPT”Row>0.33333334-:/?^T'Row> 

<-NPT> 

«Ezprassien PaititM>nad=1alaa* Simulation=*falaa*/> 

<Obaarvaiions/> 


<^oda> 

• <f4ode id-KI3t* Name=*Appropriata par Tima of Day* Typa=*Cendiiueua Imarval* ■ 
<Paranl_Noda>M9</Paranl_Noi> 

<Stalas»-{nfinity • 0.0(g0 0 • 10.0^10.0 • inlinity-:/Statas> 

• <NPT> 


<NPT Row>1.0@1.0®1.0<JNPT Rcnv> 
<NPTIRow>1 .0® 1 0@ 1 .0</NPTIRow> 
<NPT Row>1.0@1.0@1.0<yNPT Rcrw> 


<Expra»ion Paciaionad^'trua" Simulation2*trua'>M9@#Tr>an^a(30.97.5.97.5)Triangto<30.97.5J7.$)Trian9la(2.$.97.$.97.3)</Expra$aion> 
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<Obs*rvalions/> 

<Modt> 

<Notft »tf='’M34* Nam*=*Apprepri«t* pm RMponubilny* Typ»=*CenlirMjeu» lnMrv•l^• 

<P«r»nl_Nod«>M8<^ai«nl_No<t«> 

<St««s»4nf)nity • 0.0(90 0 • 10.0Q10.0 • lnfini(y</St«tds> 

• <NPT> 

<NPT Row>1.0@1 0@1 0@1.0<A«>T Row> 

<NPT Row»1.0®1 Od1.0®1.0<A«>T Row> 

<NPT Row>1.0®1 0®t 0®l.0<A«>T Rew> 

</NPT> 

<Expr»»sion Par1dion«d-*tru*' S<nitiMicn»’trut*>M8Q#Trian^»(30.97.$.78.5)Tnangl«(50.97.5^7.5)Tr)»)gl*(S0.97.S.92.S)Tnanglf(S0.97.S.97.S) 
<yExprassion> 

<Obaaivaiioni/> 

<AJeda> 

<f4o(la ■d='1IM9' Name=’AppropriaIa par Gaegraphic’ Typt=’Cenlinuoua Intarval^* 

<Paranl_N»d*>M6</Paranl_No^> 

<StMas>-(nftniiy • O.OigO 0 • 10.0^10.0 • infinjty</5taias^ 

- <NPT> 

<NPT_Row>1 .0@1 .Oei .0® 1 .0<^T_Row> 

<NPT Row>1.0®1.0®1.0®1.0</NPT Row> 

-:NPT~Row>1 .0® 1 0® t .0® 1 .0-:yNPT~Rew> 

<>NPT> 

<Ezprassion Parlilionad-trua* S(niulaliort-*trua*>M9®(9Triangla(2.S.97.S.79.7S)Triangla(12.S,97.5,929)Triangla(30.97.S.97.S)Triangla(50.97.5.97.5} 
4ExpratMn> 

«Obsaivalions/> 

■cAlada> 

■>Noda ■d=*1lM2' Name='Appropri8ta Function 02' Typa-*Continuoua Intarvat*:- 
<Paranl_Noda>M4S(9M44<'Partra_No<la> 

^StMdsMnftnjly • 0.0(90.0 • 10.0®?0.0 • lnlMy</5taio$^ 

- <NPT> 

<NPT_Row>1 .0@1.0ei.0®1 -Oei .0ei .Oei.0@1.0@l.0</NPT_Row> 

<NPT Row>1.0®1.0®1.0®1.0®1.0®1.0®1.0®t.0®1.0</NPT Row> 

<NPT'Row> 1.0®1 0®t.0®1.0®1.0®1.0®I.O®1.0®1.0</NPT~ Row> 

<iNPT> 

cEzprassion Partilionad^faisa’ SiinUblfon=‘lnia*>Arilhmatic((«1M4S)A(1iM4SHiyM44))]'M45)4<{{1/M44)A<1M4S>4(1M44)))’M44)) 
4Expra«Mn> 

<Obaarvations/> 

<A(oda> 

• <Moda id-'M43’ Nama^'Urganey Typa^tabaHatf*:- 
<Par»nl_Nodt > 

<StMa$>aetion within 10 mins®actien within 30 mns®oeiion within 60 mins®actien within 2 hn</St8tas> 

<NPT> 

<NPT_Row>0.25</NPT_Row> 

<NPT Row>0.25</NPT Row> 

<NPT’Row>0.25<AJPT”Row> 

<NPT Rew>0.25<AJPT Row> 

<AJPT> 

<Ezprassion Paitiliontd-Talsa* Simutationsfalta'/^ 

<Obsaivations/> 

<^ieda> 

<Neda id-1i444* Nama-'Appropriata par Urganey' Typa=*Continuoua tntarvar - 
<Paranl_Noda>M43«^aranl_Noda> 

<StalasMnfinity • 0.0(90 0 - 10.0® lO.O • lnfinily<)Stalas> 

. <NPT> 

<NPT Rqw»1.0®1 0®1.0®1.0<A(PT Row> 

<NPT Row>1.0®1 0®1 0®1.0<A(PT Row> 

<NPT Row>1.0®1 0®1 0®1.0<AtPT Row> 

<AIPT> 

<Exprassion Pariaionad=lrua* Simu(a(ion=tnia*>Mi3®#Triangla(90.97.S,97.5}Triang(a<S0,97.$.67.S>Triangla(t2.6.97.5,70)Triang(a(2.5,97.$.92.6) 
«yExprassion> 

<^Obsarvationa/> 

<AJoda> 

■d4odt id=’IM5’ Name='Appropriata par Cartiinty Typa='Continuout Intarval"- 
<Paran1_Noda>M46</Parart_Nodt> 

^^StalasMnlViity • 0.0(90.0 • i0.0®10.0 • ln(My</Staia«^ 

<NPT> 

<NPT_Row>1 .0® 1.0@1.0® 1 JO</NPlJR9*f> 

<NPt'Row>1 .0® 1.0@1.0® 1 i)</NPl~R9w> 

<NPT”Row> 1 .0® 1 0® t .0® 1 .0<iNPT"Row> 

<iNPT> 


cEzprassion Panitionad-trua* Simu(alion*'tnia'>M46®#Triangla<2.5.97.S.50>Tr>ang(a(30,97.5.70)Triangia(&0.97.S,87.S)Triangla<50,97.6,97.6) 
<;Cxpraasion» 
cObsaivationi/> 

<^oda> 

• <Nodaid=*M«6'Nama=-Certalnty'Typa='Lal>*lle<f'> 

<Paianl_Nftda > 

<Stalas>30% Lil(aty®S0% Ltka<y®70% (ji(ely®90% (Jkaly</Staias> 

• «NPT> 

<NPT_Row>0.25<AIPT_Row* 

<NPt3Row>0.25</NPt”Row> 

<NPT_Rdw>0.25<AIPT_Row» 

<NPt3Row>0.25</NPT~Row> 
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<Expr*ssion Paftilion»d=*f«l»«‘ Simulation=*f«l»«*/> 

<Obs»iviiions/» 

</N96»> 

^96* id='M47‘ Nsm«=*S«vtrit/ Typ«=‘Lab«lltd‘^ 

<Paitnl_Nodt > 

<Sial«»>Extraordin«(y Lift Tfwttl@Si 9 ndic«nl Lift TbrttligPQttibIt Lift Thrttl@Ex1rtordir>«ry Proptrty ThrttligSignificant Proptfly ThrttIgPostMt 
Prtttrty Thfttt< Stalts> 

• •:NPT> 

<NPT Row>0.1666e6d7</NPT Rew> 

<NPT Row>0.18e88887</T^T Row> 

-NPT Row>0.18866667</NPT Row> 

<NPT Row>O.I8666667</NPT Rew> 

<NPT'Row>0.18888887</NPt”Row> 

<NPT Row>0.18888887</T^T Row» 

<^PT> 

<Exprts«ion PMtitt&ntd=talM’ SimUatienstaltt’^ 

<Obttivtiiont/> 

■«^edt> 

• '44Qdt id='M48' Nafn«=*Apprapriatt ptr Stvtrity Typt=*Continu(Hit lnttrv«l'=> 

<Partnl_Nodt:-M47-^artn_Nodt> 

cStatts>-lnftnity • O.OtgO 0 • 10.0^10.0 • lnfMy</St«tts> 

. <NPT> 

-NPT Row>1.0@1.0(S1.0^1 0$1.0®1.0</NPT Row> 

■:NPT Row>1.0@1 0@t.Od1.0@1.0®1.0<iNPT Row> 

<NPT Row>1.0@1 0@t.Od1.0@1.0®1.0<jNPT Row> 

<>NPT> 

<Ezf>rtMion ParMitntd^'lrut'Siinul«tion=-1nit‘^M47®ffTrw)glt<70.97.5.97.8)Tntn9lt(80.97.8.97.5)Th«nglt{2.5.97.$.70)Tri«n9it(2.$.97.S.97.$) 
Tri4ng(t(2 S.97.$.70)Tritn9lt<2.$.97.$.70)c€xprt«ion> 

«Obstivtiions/> 

■«^edt> 

-ttedt id='W42 1* Namt^'Apprapriatt Function 01* Typt^'Continuout lntorvar=> 

• PartnI.Nodt-MaseiMO-Par*r<_Nodt • 
cStttts>-lnftnity • O.CKgO 0 • 10.0^10.0 • lnfMy</St«tts> 

. <NPT> 

-NPT Row>1.0@1.0(SI.0^1.0$1.i:)®1.0lSI.O®1.0®1.0<^T Row> 

<NPT Row>1.0@1.0@1.0@1.0@1.0®1.0@1.0®1.0®1.0<NPT Row> 

<NPT'Row> 1 .0^10^1.0^10^1.0@1.0@I.O@1.0@1.0</NPT~ Row> 

<>NPT> 

<Ezprtssion Partitiont^^faltt' Siniulalion2‘tn>t*>Arilh(nttic((«1M40Vl(1M40H1/M49)))'M40)-f<{<1/M48M(1/M40H1fM48)))'M48)> 
<^xprt»si«n> 

<Obteivaiiona/> 

<^odt> 

• -^Nedt id=’MS8' Namt-'Appropriatt Function 03* Typt=*Centinueu» Intarvar - 

<Partn1_Nodt>M39@M38-:/Part<il_Nodt> 

<Stalts>-tnftnily • OOtgO 0 • lO.O^To.O • lnfinily</Statts:‘ 

- <NPT> 

<NPT_Row>1.0@1 0ei.0@1.0ei.0ei Oei.0@1.0|g|1.0«/NPT_Row> 

<NPT.Row>1 .0®1 0®1.0®1.0®1 0®1 .a®1.0®1.0@t.0</NPT.Row> 

<NPT*Rovr'1 .0^1 0®1.0®1.0®1.0®1 0®I.O®1.0®1.0</NPT~ Rew> 

<#4PT> 

<Exprttsion PartiliOfW>=‘raltt* S4cnuMion=‘tnM‘>Arilhmttic(«(1M38)A(1M38)'*<1/M39)))'M38)-H{{1/M39V((1iM38>MlAI39)))'M39)> 
^xprttiiofl> 

<Obstrvalions/> 

<^odt» 

*t4odt 18=^92* Nama=’Syt Fttdback ptr Historical* Typt=*Continuout Initrvar - 
<Partnl Nodt>M31’7Partnl Nodt> 

-Slalts>4nfinity • 0.0®0 0 - 10.0610.0 - lnfMy</Stalts> 

• <NPT> 

«NPT Row>1.0@1 061.0</NPT Row> 

<NPT Row>1.0ei 0ei.0<JNPT Row> 

<NPT Row>1.061 0@1.0<JNPT Row> 

<iNPT> 

<Exprt»ion Par14iontd°trut* Siniutations*tnit*>M3169Trian9lt<2.S.97.$.$0)Triangtt<2.$.97 M7.S)Triangit(2.$.97.5.$0)<^xprttsion> 
<Ob$ttvaiion«/> 

■<^ode> 

<t4odt id-*M53' Namt=’Syt Fatdback ptr Rtaltimt' Type=‘Coiitinuoua Inttrvar * 

<Partnl_Nodt>M32</P8rtrt_Nodt> 

<Stalts>-{nfinity • 0.060 0 • 10.0610.0 • lnfinily:/Stalts> 

• <NPT> 

<NPT_Rew> 1.061061-061061 .0<JNPT_Rew> 

<NPT Row>1.061061061061 0<^PT Row> 

<NPT~Row>1.061061061061 0<^PT~ Rew> 

<iNPT> 

<Exprt»ion Partiliontd^'tnJt' SiniulaHons'tnit*>M326in’rianglt(2.8.87.5.40)Trianglt(2.$.87.9.80)Tri8nglt(2.8.97.5.70)Tri8nglt(2.5.97.9.87.8)Trian9lt 
(S0.97.S.97.5>c£xprtS3ion> 

<Obttivalioni/> 

<^odt> 

<No(tt »d='M33* Namt=*Accuracy ptr Lecatien' Typt=*Continuout Inttrval*^ 

<Partnl_Nodt>M34</Partr4_Nodt> 

<Stalts»-tnfinity • 0.060 0 • 10.0610.0 • inlinily:/Statts> 

• <NPT> 
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<NPT_Row>1.0®1 0©1.0<^PT_Row> 

<NPT Row>1.091 OQI.OONPT Row> 

<NPT”Row> 1 ,0®1 0©1.0<NPT“Rew> 

<#JPT> 

<Exi>r*Mion Par1<i<>n*d='tru*' Simulation=*lni*‘^M34©#Th«>gl«{2.5.97.5.S0)Thang4<S0.97.S.76.7S)Thwgl«(l2.5.97.5.50)</Expf«ssion» 
<Obstrvalions/> 

• <Nod« id=’M$f* Nim»=''Accur«cy p«r Typ«=''Continuout lnt«rv«r> 

■ Par»nl_h4od* *M3}-^Par*n_Nod* - 

•:Stal«s>>lnAnily • 0.0©0 0 • 10.0©10.0 • lnAniy</Stalts> 

• <NPT> 

<NPT Row>1.0eiO©1.0<rt»lPT Rerw> 

<NPT.Row>1 .0®1 0©1.0<NPT.Rerw> 

<NPT'Row> 1 .0@1 .0@i .0«yNPT*‘Rerw> 

<^PT> 

<Expr»uion Paditionad-tni*' S«nul«tior)siriw^>M33©in'nan9l«<30.97.5.97.5)Tn«ngl»{$0.97.5.67 .5)Th«ng(«(2.$.87.5.40)</Exprt«»ion> 
<Ob$»tvaiion«/> 

• -«t4o<t« id^'MST’ Nam«-*Efr«ctiv9 par Timalinta*' Typ»='Coiitinuou» Intarval'^- 

<P«rtnl_Npd»>M23</Partrt_Nodt> 

cStat*s>-trAnity • 0.0©0 0 • To.0©10.0 • lnlMy</St«t*s> 

• <NPT> 

<NPT Row>1.0©1.0©1.0©1.0</NPT Row> 

--NPT’Row>1 .0©1 .0@1 .0© 1 .0<#^T’Row> 

■:NPT Row>1.0©1 0©!.0©1.0 tA^T Row> 

<>NPT> 

cEzprassion Parlitionad-trua* Simulalion=‘tnM*:-M23©#Triangla(2.5,67.S.d7.5)Triangla(2.5,97.S.97.S)Triangla(2.S.97.S.67.S)ThaAgla<2.5.97.S,70) 
cJExpra«sion> 

<Ob«*iv«iion«/> 

<^ada> 

• -t4oda id=‘M59' Nama='Eff«ctivt par PubNc Awaranaa*' Typa^'Conbinuoita lntarval*=>^ 

«:Paranl_Npda>M24</Paftrt_Nod#> 

cStalas>-lrAnily • 0.0©0 0 • 10.0©10.0 • lnlMy</Statas> 

• <NPT> 

<NPT Row>1.0©1.0<iNPT Row> 

<NPT’Row> 1 .0©1.0<d4PT” Row> 

<NPT Row>1.0©1.0<4<IPT Row> 

<>NPT> 

cEzprassion Parlilianad-*trua* Sanulation='lrua'^-M24©fTriangta<2.S.97.S.97.S)Triangla<2.S,97.5.67.S)</ExprassiM> 

<Ob$arvaiions/> 

<Mbd4> 

- ‘ttode Name==*Effactive par Atert Frequefiey* Type='Cantinuaua Interval'^- 

<Paranl_N»dt>M26</Paranl_Noda> 

<Stalas>-{nfinity • 0.0©0.0 • 10.0©10.0 • lnfinity</^alas> 

• <NPT> 

<NPT_Row>1 .0@1 0® 1 .0<.iNPT_Rew> 

<NPT Row>1.0©1.0©1.0<^PT Raw> 

<NPT>ow>1 .0@1 -Oei .0«^PT''Rcrw> 

<yNPT> 

<Exprassion Part4K>nad=*trua* Simulatton-*trua‘^‘M26©9Triangta(30,97.S.97.S)Tflangla{50.97.S.97.5)Triangla<30.97.5.97.S>‘:/Exprassien> 
<Obsarvalions/> 

<Modt> 

• <Noda id=*M62' Nama=XonibirMtion 01* Typa=*Continuoiit Intarval’^ 

<Parant Noda>MS7©MS9</Parard Noda> 

<S(ala(>-lnfinily • 0.0©0 0 - 10.9©To.0 • lnAnily</$lalts> 

• <NPT> 

<NPT Row>1.0©1 0©t 0©1.0©l.0©1.0©l.0©1 0©1.0</NPT Row> 

<NPT Row>1.0©1 0©1.0©1.0©1.0©1 0©1.0©1 0©1 0<yNPT R9W> 

<NPT Row>1.0©1 0©).0©1.0©1.0©1.0©I.O©1.0®1.0</NPT Row> 

<WPT> 

cExpraasion Partitiof>ad=’falaa* Stniulalion-*lrua*>Arilhmatic((((1M57)A(1M$7)4<1/M$9)))'M$7)4^({1/M$9)A(1A457)*<1AIS9)))'M$9)> 
'^xprassion> 

<Obsaivaiioni/> 

<^eda> 

• 44ode id='M03' Nama^'CombinatMn 02’ Typas'ConlinuoiM Intarval'- 

<Paranl N»da:-M61©M2S-wParem Nede^ 

<Slata«>-{nfinfly • 0.0®0.0 • 10.0©?0.0 • lnfiniy</Stala$> 

. «NPT> 

<NPT Row>1.0©1 0©1.0©1.0©1.0©1 .&<jMPT Row^- 
<NPT Row>1.0©1 0©t.O©1.0©1.0©1.0<^PT Row> 

<NPT_Row> 1.0©1 0©1.0© 1.0©1.0©1 .0<iMPT_Row> 

<INPJ> 

<Ezprassion Partilionad=’fa(aa‘ S4mi4aiion=*lrua*>Anlhmatic((«lJM25)/((1M25)4<1/M61)))'M2S)4<«iyM6l)/((tM2S>4<t/M6l)))’M61)) 
';CxprassMn> 

<Obaarvaiions/> 

■cModa> 

• <Noda id=*M(4‘ Nama^’Cembinalien 03* Typa=*Con(inupua lntarval'> 

<Par*n1_Noda>M62©M63<'Parara_Noda> 

■^StalasMnfinlly • 0.0©0 0 • 10.0©To.0 • lnfinily:/Stalas> 

. <NPT> 

<NPT_Row>1 .0©1 0©t .0©1.0©1.0©1.0©l.0©1 .0©1.0</NPT_Rew> 

<NPT Row>1.0©1.0©1.0©1.0©1.0©1.0©1.0©1.0©1.0</NPT Row> 
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<NPT Rew>1 .0^1 0^10^1 0®10®1.0e(I.O®1.0ei O<VNPT Row> 

<JNPT> 

<Expr*Mion Partilion«d=fal««‘ &muMii>n=‘lnM*>Aril»m*tie(«(1Me2)A(1M62)4<1>M63)))‘M62)4<({1/M$3M((1A462)«<1M63)))’M63)) 
<£xprassion> 

<Obs*rvaiions/> 

• <NQd* •d=*Mdi* N«m«='EffMtiv« p«r Aftw Action* Typ«=*Con(inueM* lnl«rval*> 

<P«r*nt_Nod*>M27«/Par«nl_Nod«> 

<St 0 l«$>-lnftnily • O.OQO 0 • io OQIO.O ■ lnfinity</St«lts> 

• <NPT> 

<NPT Row>1.0®1 0@1.0«yNPT Row> 

-NPT Row>1.0eiO®1.0<rt^PT Rerw> 

<NPT Row>1.0®1 0®t.O<NPT Row> 

<AIPT> 

<Ezpr*ssion Par1ilK>n*d-*lriJO* Simulation=*truo*>M27®#Triangil«{2.S.97.5.50)Tnangl«<^.97'&70)Triongl*(S0.97.5,87 .S>«£x|9r«ssion> 
<Ob»»rv«lion«/> 

<Aid6*> 

- -^^ede •d^'MM' Name='EfF«ctive per Accureey* Type=*Continuetis Intervel*:- 

<P«r*n1_N»de>M55l9M$6<'P«rfrt,Nod*> 

<Sl«t««>'{nnnity • 0.0®0.0 • 10.0®To.O • lnfiniy</St«tt«> 

• <NPT> 

<NPT_Revr- 1.0® 1 0® 1.0® 1.0® 1.0® 1.0® 1.0® t. 0® 1.0 </NPT_Row> 

<NPT Row>1.0®1.0®1.0®1.0®1.0®1.0®1.0®1.0®1.0</NPT Row> 

--NPT'Row> 1 .O®1.0®t.0®1.0®1.0®1.0®l.0®1.0®1.0</NPT~ Row> 

<iNPT> 

^Expression Partilioned-*fatoe‘ SiinUlaiion*true*>Arilhmetic((«1M55)/((1iM55)4<1/M58)))*M55)4<((1Ail58)/«1iM55>«<1AI56)))'M56)) 
•VExpression> 

<Ob$ervaiions/> 

<Mode> 

• ->740(10 »d-‘IM7’ Name=*E(foctive par Undarstandability' Typa^XontiniMus Interval':- 

<Parenl_Node>M29</Psr«nl_Node> 

<Stata$>'lnfinity • 0.0®0.0 • 10.0®10.0 • lnfiniy</^ate$> 

• <NPT> 

<NPT_Rovr:- 1.0® 1 0®) .0® 1 .0<AIPT_Row> 

<NPT Row>1.0®1.0®1.0®1.0</NPT Rew> 

--NPT’Row>1 .0®1.0@1.0® 1-0</NPT’Row> 

<yNPT> 

cExpression Panitioned-lrue* Simulalion-'tnie*:-M29®#Triangle(2.5.97.5.70>Triangla(2.5.97.5.87.5)Trianglo(2.5.97.5.97.5)Triangle(2.5.97.5.97.S} 
•VExprossion> 

<Ob$ervations/> 

-<Mode> 

- -t4ede id°'M88' Name-’Effeetive per Public Feedback* Type^'Continuoua Interver:- 

<Parenl Nod»>M30</Par«rd Node> 

<Stalas>-lnfinity • 0.0®0.0 - io.0®10.0 • lnfiniy</Stalts> 

• <NPT> 

<NPT_Row>1 .0® 1 0® 1 .0<NPT_Row> 

<NPT Row>1.0®1.0®1.0<^PT Raw> 

<NPT>ow>1 .0@1.0®! .0«:JNPT''Rcrw> 

<yNPT> 

<Expression PartdK>ned=*true* Simulatton-*true‘:‘M30®9Triaftgta<50.87.S.97.S>Tnang|e{50.97.5.87.5)Triangle<70.97.5.97.S>‘:/Expression> 
<Observalions/> 

<Mode> 

• <Noda id=*M59' Name=Xembination 04* Type=*Centinueiit Interval’^ 

<Parent Node>M68®M65</Parenl Node> 

<S(alas>-lnrinily • 0.0®0.0 - 10.0®To.O • lnAnily</$lalts> 

• <NPT> 

<NPT Row>1.0®1 0®t 0®1.0®l.0®1.0®l.0®1 0®1.0</NPT Row> 

<NPT Row>1.0®1 0®1.0®1.0®1.0®1 0®1.0®1 0®1 0<^T Row> 

<NPT Row>1.0®1 0®I.O®1.0®1.0®1.0®I.O®1.0®1.0</f4PT Row> 

<#4PT> 

<Expression Partition«d=’rai«e* Stmulalion^tnM'‘>Arilhmetic((((1M86)A(1M65)4<1/M66)))'Me5)-H«1>M66)A(1/M65>e<1/M68)))'M66)> 
';£xpression> 

<Obseivaiioni/> 

<44ode> 

• <Node id=’M70' Name^Xombinaiien 05’ Typa^'ConlinueiM Interval':^ 

‘:Parenl_N»de:-M68®M67-:Perefit_Nede:- 
<Slala$>-{nfinity • 0.0®0.0 • 10.0®To.0 • lnfiniy</Stale$> 

. «NPT> 

<NPT Row>1.0®1 0®1.0®1.0®1.0®1.0®I.O®1.0®1.0-c/NPT Row> 

<NPT_Row>1.0®1 0®1.0®1.0®1.0®1.0®l.0®1.0®1.0<:/NPT_Row> 

<NPT_Row>1.0®1 0®1.0®1.0®1.0®1.0®l.0®1.0®1.0-c/NPT_Row> 

<44PT> 

<Expression Partilioned=’faiee‘ S4mi4aiion=*lrut*>AriUvnetic((((1AM7)/((1M67)4<1/M68)))'M67)4<«iyM68)/((trM67>4<t/M68)))’M68)) 
';Cxpression> 

<Obs»rvaiions/> 

■c44ode> 

• <Node id=*M71* Name^’Cembinalien 06* Type=*Centinuous Interval'^ 

<Parenl_Node>M69®M70<'Parerd_Node> 

■':Stalas>-{nfinlly • 0.0®0 0 • 10.0®To.0 • lnfinity:/Stales> 

. <NPT> 

<NPT_Row>1 .0®1 0®1.0® 1.0® 1.0®1.0® l.0®1.0® 1.0-c/NPT_Row> 

<NPT Row>1.0®1.0®1.0®1.0®1.0®1.0®1.0®1.0®1.0-^T Row> 
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<NPT Rew>1 .0^1 0^10^1 0®10®1.0e(I.O®1.0ei O<VNPT Row> 

<JNPT> 

<Expr*Mion Partilion«d=fal««‘ &muMii>n=‘lnM*>Aril»m*tie((«1Me9)A(1M69)4<1>M70)))‘M69)4<((1/M70M((1M69)«<1AI70)))’M70)) 
<£xprassion> 

<Obs*rvaiions/> 

<A^9d*> 

<NQd* 14=1472* N«m«='AvailaM« pm Sy«t*m_g12' Typ«='CofltinMOU» lnt*rvar> 

<P«r*nt_Nod*>M11«/Par«nl_Nod«> 

<St«l«$>-lnftnily > O.OQO 0 • io OQIO.O ■ lnfiniy</Stalts> 

• <NPT> 

<NPT Row>1.0®1 0@1.0«yNPT Row> 

-NPT Row>1.0eiO®1.0<rt^PT Rerw> 

<NPT Row>1.0®1 0®t.O<NPT Row> 

<AJPT> 

<Expr*ssion Par1ilH>n*d=*lrij** Simulation=*tru«‘>M11®#TrMngt«<S0,97.S.97.S)Trwngla(S0.97^.d7.S)Tnangk<S0.d7.S.97.5)</ExprMaion> 
<Ob»trv«iion«/> 

<Mdd«> 

-44ede ^=1473* Name^'Availabl* pec System_®13' Type^'Cantinuous lntarvar> 

<Par*nl_Npdt>M13</P«r*nl_Nod*> 

<St«t««>'lnnnity • 0.0®0.0 • 10.0®10.0 • lnfiniy</9alt«> 

• <NPT> 

<NPT_Row:-l .0®1 0®1.0<NPT_R<iw> 

<NPT Row>1.0®1.0®1.9<^PT R9W> 

--NPT’Row> 1 .0®1.0@1.0<NPT’Reiw> 

<iNPT> 

cExprassion P8rlition«d='trua* S)niuMion=*tnM*>M13®fTri«ngla(S0.97.S.97.5)Tnangl«<2.5,97.S,97.&}TriAng(«<2.5.97.S.97.S)<£xpr«ssion> 
<Obsaivalioni/> 

•^edt> 

"l^oda ids’M74* NamaB'Availabla par Eaaa of Uaa* Typa='Centinuout lntarvaJ'> 

• Param.Noda>M81 @M87 >Par*r«_No<tt> 

<S(Blas>'{nfinity • 0.0@0.0 • 10 0®10.0 • lnfiniy</^atts> 

. <NPT> 

<NPT_Row>1 .0®1 0®1.0®1.0®1.0®1.0®l.0®1.0®1.0</NPT_Row> 

<NPT_Rew:-t.0®1 0®1.0®1.0®1.0®1.0®l.0®t.0®1.0</NPT_Row> 

<NPT Row>1.0®1.0®1.0®1.0®1.0®1.0®1.0®1.0®1.0<7NPT Row> 

</NPT> 

<ExprtMion Partition«t=-rolaa' S<mUalion^-tn»a‘>Aritfmttic((«1M81)/l(1MeiH1/M87)))'M81M((1/M87V((1/M81H1Me7)))*M87)) 
•;£xprassion> 

<Obsaivalioni/> 

•^edt> 

"l^oda »d«'M75’ Nama^'AvailaMa par Syatam Raliabilit/ Typa=‘Con(inuoua Intarval'^ 

'Parenl_N»de>M 14 •VParerl_Node> 

<Stalas>-lnfinity • 0.0@0.0 • 10.0®10.0 • lnfinity</^alts> 

- «NPT> 

<NPT Rdw> 1.0®1 0®1.0>-VNPT Row> 

<NPT_Row>1 .0@1 0®! .0<JNPT_Row> 

<NPT Row>1.0®1.0®1.0<vNPT Row> 

<Exprassion Par14H>nad='trua'' Simulation=‘tnia‘>M14®lin'hangla(70.97.S.97.$)Thangla(70.97.S.97.5)Tnan9la<80.97.S.97.$)</Expratsion> 
<Obsarvalions/> 

</No6»> 

<Notfa id=*M7l* Nama=*Avajlabla par Training' Typa=Xen(iniMua Intarval’^ 

<^Paranl.Nada>M84®M69«Paraf4.No4a> 

<Stalas>4nftnilv • 0.0®0 0 • 10 0®?0.0 • lnfinity</St«las> 

• <NPT> 

<NPT Row>1.0®1 0®1 0®1.0®1.0®1.0®1.0®1.0®1.0«/NPT Row> 

<NPT Row>1.0®1 0®1 0®1.0®1.0®1.0®I.O®1.0®1.0<4^T Row> 

<NPT Row>1.0®1 0®1.0®1.0®1.0®1 0®1.0®1.0®1.0<7NPT Row> 

<4JPT> 

^Exprassion Partitiona<l=’raisa’ S<niulalion=’lnia'>Anlfmatic((((1M84V1((1MB4)4<1/M88))}'M84H((!/M88yi((1/M84)4<1/Me8)))'M88)) 
<>Exprastion> 

<Obaarvaiions/> 

<Moda> 

■<t4oda id=*M77* Nama^'Availabla par Sacurfty* Typt=*Continu«u» lntarvar> 

«Parani_Noda>M 1 $</Parart_Noda> 

'cStalas-'-tnlSnily - 0.0®0 0 - 10.0®10.0 - lnlMty^.Stataa^ 

• <NPT> 

<NPT Row>1.0®1.0®1.0®1J)<4«>T Row> 

<NPT~Row>1 .0® 1 0® 1 .0® 1 .0<A«>T~Row> 

<NPT Row>1.0®1 0®t.0®1.0-:4«>T Rew> 

<iNPT> 

<Ezpra»ion Pa(titM>nad=lfua' S<mulation='lrua'>M16®irrnan9la<50.97.5.97.5)Triangla(30.97.$.87.5)Trian9la(2.$.87.5.70>Triang(a(2.5.87.8.90) 
'^xprassMn> 

<Obsaivaiioni/> 

<^eda> 

• <Noda id=*M78' Nania=Xen8>in«(ien 09' Typa=*Con(inuoua lntarval‘> 

<Paranl N»da>M72®M73<JParanl Noda> 

<SMa«>-{nfinity • 0.0®0.0 • 10.0®To.0 • lnfiniy</Stala$> 

• <NPT> 

<NPT Row>1.0®1 0®1.0®1.0®1.0®1.0®1.0®1.0®1.0<.'NPT Row> 

<NPTIRow> 1 .0®1 0®1.0® 1.0® 1.0®1.0® l.0®1.0® 1.0<4«IPTlRew> 

<NPT Row>1.0®1.0®1.0®1.0®1.0®1.0®1.0®1.0®1.0<.'NPT Row> 
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<^IPT> 

<Expr*ssion Partilion«4=fato*‘ &niUalion=‘tnM*>Ariltm*tic<«(1M72)/((1/M72)4<1/M73))]'M72)-H((1/M73M{1AII72>«<1M73)))’M73)) 
<;Cxprtstion> 

<Obstivilions/» 

• <Ned« id='M79* Nam«=Xoinbinatioii Ot* Typ*=*C«iNinuein lnt*rv^‘> 

<P«r*nl_Nod»>M74QM7S<'P«r«f<_Nod«> 

<Sl«l«s>-lnfinity • O.OQO 0 - IO.OQTo.O • lnfinity</St«lts> 

• <NPT> 

<NPT Row>1.0d1 0^1.0^1.061.0^1.0(^1.061.0®1.0</t^T Rew> 

<NPT Row>1061 06^ 061 061061.061.061.061 O-c.'NPT Row> 

-NPT Row>1.061061061061061061061061 0</NPT Row> 

<iNPT> 

<Expr*ssion Partilioncd-fal**’ S<(niialion**lnia‘>Anttimatie((«1iM74)A(1iM74)4<1/M75)))*M74)4<((1/M75)/H1/M74>«(1AI75)))*M75)) 
<^xprassign> 

<Ob»»rv«lion«/» 

<Aid6*> 

- ■'>Jode id-’MbO* Name=Xombinatian 07* Typ«-'CoMinuous Intarvd* • 

<Paf»nl_N«»d*>M706M77</Paffrt.Nodt> 

<Sta1a«>'{nfinity • 0.060-0 • 10.0610.0 • lnfiniy</9alts> 

• <NPT> 

<NPT_Rew:-1.06106106106106106' 061 0610</NPT_Row> 

<NPT Row>1.061 061 061 061 061061 061 061 0</NPT Row> 

--npt'row>i .061061061061061061061061 o</npt~ Row> 

<iNPT> 

cEx^assion Partilionad^fatoe' Siini4aiion»*trua*>Arilhmatic((«1M76)A(1iM76H1/M77))]-M76)4<({1/M77)A<1M76>4<1M77)))'M77)) 
•VExprassMn> 

<Ob$»rvaiions/> 

<Mdd4> 

• •‘t'ioda id-‘M01’Name-*Eaaa ofUsa par Magnitude' Type^'Cantinuaua Intervar- 

<Paranl_Nada>M17</Paranl_Noda» 

<Statfs>'lnnnity • 0.060-0 • 10.06100 • lnfiniy</9atts> 

• <NPT> 

<NPT_Revr:-1 .061 061 061-0-:A^T_Rew> 

<NPT Row>1.061 061 061-0</NPT Rew> 

<npt’row>i .061061 061-0</npt’row> 

<yNPT> 

<:Expression PaitiUenad-true* Simulation-*1riia*>M176l*^nangla<S0.97.5.d7.S}Tflangle(12.5.97.S.87.S)Triangla(2.S.97.5.50)Triangle(2.S.97.S.30) 
•VExprassian> 

<Obsarvations/> 

<Mdd4> 

- -44ede>d°‘M82‘Name^Eaaeof Uaapar Craas S^lem' Type^'Cantinueualntarval'^- 
<Pafanl_Nada>M19</Parart_Nod#> 

<Stalas>-lnfinity • 0.060 0 • 10.0610.0 • lnfinity</^alts> 

• <NPT> 

<NPT_Row>1 .061 -0<-NPT_Row> 

<NPT Row>1 .061 0<A4PT R9W> 

<NPT>ow>1.061 0 <^PT'Row> 

<yNPT> 

<Exprassien Part4ionad=*trua* Sinnulalion-*lrua*>M16611i1'fiangla(&0,97.S.79.7S)Triangla{S0.97.5.97.&)<£xpress*on> 

<Obsarvalions/> 

<A«lede> 

• <Noda id=‘M83* Nama=^iaa of Uaa par Tamplata* Typa=*CarMinueut Intarvar^ 

<Parant Noda>M19*^aranl 

<S(ala»>-infinj|y • 0.060 0 - io.0610.0 • lnAnly</Slalts> 

• <NPT> 

<NPT Row>1.061 0<A>»PT Raw> 

«NPT Row>1.0610<A^PT Rwr> 

■:NPT Row>1.061 0</NPT Raw> 

<WPT> 

cExpraasion Pstiaionad^'tnja’ Sanulation=lri»a*>M196ll'1'riangta<2.5.97.S.97.S)Triangla<2 S,97.5.87.S)<Exprass>on> 

<Obsaivaliona/> 

<Mada> 

• ^oda id='M84‘ Nama^'Training par SIdl' Typa^Xontinuauf Interval'^ 

‘:p»fani_l4oda>M20</Parart_Noda> 

'cStalas-'-trrftnily - 0.060 0 - 10.0610.0 - lnfinity^.Stataa^ 

• <NPT> 

<NPT Row>1 .061 061 OOMPT Rcrw> 

<NPT”Row> 1 .061 061 -0-vNPT~Row> 

<NPT Rew>1.061 061 0<NPT Rew> 

<yNPT> 

<Exprt»ion Partitionad^trua' S*nulation2'tnia*>M2O61Triangla{3O.97.5.97.5)Triangla(SO,97.$.67.9)Triangla0O.97.S.7O)<yExpraniQn> 
cObaarvaiioni/> 

<^oda> 

• <Noda id='M8$‘ Name=Trainir»g par Undaratandng' Type=Xon(inuoua lnlarvd'=> 

<Paranl_Nfrda>M2 1 ■^arart_Noda> 

<Stalas»-{nfinity • 0.060 0 • 10.0610.0 • inMty-:/Statas> 

• <NPT> 

<NPT_Row>1 .061 061 -0</NPT_Row> 

<NPT Row>1 .061 061 O'VNPT Rcrw> 

<NPTIRow> 1 .061 061 -0<NPTlRew> 

<JNPT> 
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<Expr«ss*on ParMion«d=‘tni«* Sifflulat)on-*triM*>M21®*Trian9l*(S0.97.$.97.5)Tri8ngl«(2 S,97 S.70>Triangita(2.9.97.S40)<>Exprasiion> 
cObsaivaiions/> 

• «Notfa id=^M* Nama=‘Training par Praatka* Typa=*Continueua lntarval*> 

<P8rant_Noda>M22</Paranl_Noda^ 

<Stala$>-4nftrHtv • O.OQO 0 • 10.0Q10.0 • lnfMty</Slaias> 

• <NPT» 

<NPT Row»1.0®1 OdKOvNPT Row> 

•:NPT.Row>1 ,0®1 0®1.0<JNPT.Row> 

<NPT Row>1 .0®1 0®t.0</NPT Row> 

<*IPT> 

<Exprtssion ParWionad-trua* Siniiulation-'tnM*>M22®9Tnangla(2.5.97.5.67.S)Triangla(2.S.97.5.67.S)Triengla<2.5,97.5.S0)</Exprassion> 
<Obaaivaiiona/> 

<jt4oda> 

- <Noda id»'M97' Nama=*Ceinbinatian 19* Typa-'Ceminuain lntarvd’> 

<Paranl_Noda>M92®M83‘'Partra_Noda> 

<StMas>-4nftniiy • 0.0®0 0 • 10.0®To.0 • ln(injty</5taias> 

- <NPT> 

<NPT_Row>1 .0@1.0ei.0@1 -Oei .Oei .Oei.0@1.0@l.0<7NPT_Row> 

<NPT Row>1.0®1.0®1.0®1.0®1.0®1.9®1.0®1.0®1.0<NPT Row> 

<NPT'Row> 1 .0®1 0®1.0®1.0®1.0®1 .O®l.0®t.0®1.0</NPT~ Row> 

<INPJ> 

<Ezprassion Partilianad-faiaa' SiinUklion-*trua*>Arilhmatic((«1M82)A(1iM82)4<1/Md3))rM92}4<({1Ail83)A<1M82>4<1M93)))’M83)) 
4Exprat»«n> 

«Obsaivalions/> 

<^4oda> 

• ■:Neda id-'M99' Nama^'Cofnbinatian 11* Typa-'Cantinuaua Intarvd':- 
<Par»nl_Noda>M89®M86<'P8rara_Noda> 

<SlM«$Mnfinj|y • 0.0®0.0 • 10.0®i0.0 • lnMty</5tai*$^ 

• <NPT> 

<NPT_Row>1 .0@1.0ei.0@1 -Oei oei .0@1.0ei oei 0 <^T_Row> 

<NPT Row>i.oei oei.oei oei.oei oei oetoeio*^''’ 

<NPT~Row>i.oei oei oei.oei.oei oei oei oei o</npt~row> 

<iNPT> 

cEzprassion Partilionad^faisa' Siinuiaifon=‘lnia*>Arilhmatic((«1M8S)A(1iM85)4<1/Md6))rM85)4«(1/M86)A<1M8S>4<1M86)))*M86)) 
4Exprat»«n> 

<Obaarvations/> 

<^itoda> 

<At8fc Objad> 

</Risk_Modat> 
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Appendix U Public Trust Model Validation Scenarios and Results 


Scenarios 

Factors 

Responses 

# 

Cases 

Target 

Node 

Scenario Description 

30 Explain what has happened 

70 Why I should act 

23 Who should act 



<, 

CO 

u 

Q 

LU 

u. 

CD 

X 


1 

Case 1: Includes 
"what,” Includes "why,” 
omits “who” 

Relevance 

“Nuclear plant radiation release 
in this area 

Take shelter now 

to avoid radiation exposure” 

How likely are you to consider this alert to be relevant to you? 

1 

1 

0 



6 

4 

4 

7 

3 

6 

3 

7 

7 

Case 2: Includes 
"what,” omits “why,” 
includes “who’ 

Relevance 

“Nuclear plant radiation release 
in ZIP 12345 

Take shelter now” 

How likely are you to consider this alert to be relevant to you? 

1 

0 

1 



6 

6 

5 

7 

5 

7 

5 

5 

7 

Case 3: omits “what,” 
includes “why,” in¬ 
cludes "who” 

Relevance 

“Nuclear power plant warning 
in ZIP 12345 

Take shelter now 

to avoid radiation exposure” 

How likely are you to consider this alert to be relevant to you? 

0 

1 

1 



5 

7 

4 

7 

5 

7 

5 

7 

7 
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Scenarios 

Factors 

Responses 

# 

Cases 

Target 

Node 

Scenario Description 

30 Explain what has happened 

70 Why 1 should act 

23 Who should act 



<, 

CQ 

o 

a 

LU 

u. 

CD 

X 


1 

Case 4: omits “what,” 
omits “why,” omits 
“who” 

Relevance 

“Nuclear power plant warning 
in this area 

Take shelter now” 

How likely are you to consider this alert to be relevant to you? 

0 

0 

0 



4 

2 

3 

7 

4 

6 

2 

5 

7 
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Scenarios 

Factors 




Responses 







# 

Case 

Target 

Node 

Scenario Description 

10 Action to take 

24 Time window to act 

32 Lead time provided 

23 Who should act 

1 Relevance 

< 

CQ 

o 

Q 

LU 

u. 

O 

X 



Case 1: defined action, 


“Flash flood in ZIP 12345 from 4:55 to 5:35 PM Evacuate low 
lying areas” 
















defined time, sufficient 
lead, includes “who,” 
irrelevant 

Acting 

You determine that the alert is not relevant to vou (e.a., it 
does not apply to your location, it is not issued in a timely 
manner, or it does not address an emergency that affects 
you). 

1 

1 

1 

1 

0 

2 

3 

2 

1 

2 

1 

1 

1 

2 


Case 2: defined action, 
defined time, insuffi¬ 
cient lead, omits 
“who,” relevant 

Acting 

“Flash flood 
in this area 
from 4:40 to 5:20 PM 

Evacuate low lying areas” 

1 

1 

0 

0 

1 

6 

6 

5 

7 

5 

7 

6 

7 

7 

2 

You determine that the alert is relevant to vou (e.a., it aoDlies 
to your location, it is issued in a timely manner, and it ad¬ 
dresses an emergency that affects you). 
















Case 3: defined action, 
undefined time, suffi¬ 
cient lead, omits 
“who,” relevant 

Acting 

“Flash flood 
in this area 
at 4:55 PM 

Evacuate low lying areas” 

You determine that the alert is relevant to vou (e.a., it aoDlies 
to your location, it is issued in a timely manner, and it ad¬ 
dresses an emergency that affects you). 

1 

0 

1 

0 

1 

7 

6 

5 

7 

6 

7 

6 

7 

6 
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Scenarios 

Factors 

Responses 

# 

Case 

Target 

Node 

Scenario Description 

10 Action to take 

24 Time window to act 

32 Lead time provided 

23 Who should act 

1 Relevance 

< 

CQ 

o 

Q 

LU 

u. 

o 

X 



Case 4: defined action, 
undefined time, insuffi- 

Acting 

“Flash flood 
in ZIP 12345 
at 4:40 PM 

Evacuate low lying areas” 

1 

0 

0 

1 

0 

1 



1 

1 

1 


1 



cient lead, includes 
"who,” irrelevant 

You determine that the alert is not relevant to vou (e.a., it 
does not apply to your location, it is not issued in a timely 
manner, or it does not address an emergency that affects 
you). 

7 

2 

2 

2 

2 

Case 5: undefined 
action, defined time, 
sufficient lead, omits 
“who,” irrelevant 


“Flash flood 
in this area 

from 4:55 to 5:35 PM” 
















Acting 

You determine that the alert is not relevant to you (e.g., it 
does not apply to your location, it is not issued in a timely 
manner, or it does not address an emergency that affects 
you). 

0 

1 

1 

0 

0 

1 

2 

2 

1 

1 

1 

3 

1 

2 


Case 6: undefined 
action, defined time, 

Acting 

“Flash flood 
in ZIP 12345 
from 4:40 to 5:20 PM” 

0 

1 

0 

1 

1 











insufficient lead, in¬ 
cludes “who,” relevant 

You determine that the alert is relevant to vou (e.a., it aoDlies 
to your location, it is issued in a timely manner, and it ad¬ 
dresses an emergency that affects you). 

5 

7 

4 

7 

6 

7 

6 

7 

7 
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Scenarios 

Factors 




Responses 







# 

Case 

Target 

Node 

Scenario Description 

10 Action to take 

24 Time window to act 

32 Lead time provided 

23 Who should act 

1 Relevance 

< 

CQ 

o 

Q 

LU 

u. 

o 

X 



Case 7: undefined 
action, undefined time, 

Acting 

"Flash flood 
in ZIP 12345 
from 4:55 PM” 

0 

0 

1 

1 

1 










2 

sufficient lead, in¬ 
cludes "who,” relevant 

You determine that the alert is relevant to vou (e.a., it aoDlies 
to your location, it is issued in a timely manner, and it ad¬ 
dresses an emergency that affects you). 

7 

6 

5 

7 

7 

7 

6 

7 

7 


Case 8: undefined 
action, undefined time, 

Acting 

"Flash flood in this area at 4:40 PM 

You determine that the alert is not relevant to vou (e.a., it 

0 

0 

0 

0 

0 

1 



1 

1 

1 


1 



insufficient lead, omits 
"who,” irrelevant 

does not apply to your location, it is not issued in a timely 
manner, or it does not address an emergency that affects 
you). 

2 

5 

2 

2 
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Scenarios 

Factors 

Responses 

# 

Case 

Target 

Node 

Scenario Description 

99 Type of alert 

7 Frequency 

3 Public awareness of WEA 

20 History of relevance 


< 

CQ 

o 

Q 

LU 

u. 

C3 

X 





The alert pertains to an imminent threat (i.e., a flash flood) in 
your area 
















Case 1: imminent 
threat, few prior alerts, 
sufficient public out- 

View as 

Over the oast 6 months, vou have received few alerts - 6 from 
the National Weather Service regarding severe weather 
events, 1 AMBER alert, and 2 alerts from your city’s EMA 
regarding local threats. 

1 

1 

1 

1 


2 

6 

3 

1 

2 

2 

2 

1 

2 


reach, history of rele¬ 
vance 

Spam 

You have oreviouslv been made aware of the WEA orooram 
via public outreach (e.g., newspaper articles, mobile service 
provider mailings, TV and radio news stories) from the EMA. 















3 



You believe that most of the prior alerts that you have re¬ 
ceived were relevant to you. 

















The alert pertains to an imminent threat (i.e., a flash flood) in 
your area 
















Case 2: imminent 
threat, few prior alerts, 
insufficient public out- 

View as 

Over the oast 6 months, vou have received few alerts - 6 from 
the National Weather Service regarding severe weather 
events,) AMBER alert, and 2 alerts from your city’s EMA 
regarding local threats. 

1 

1 

0 

0 


4 

4 

5 

1 

5 

5 

5 

6 

2 


reach, history of irrele¬ 
vance 

Spam 

You have not been made aware of the WEA orooram via 
public outreach (e.g., newspaper articles, mobile service 
provider mailings, TV and radio news stories) from the EMA. 


















You believe that most of the prior alerts that you have re¬ 
ceived were not relevant to you. 
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Scenarios 

Factors 

Responses 

# 

Case 

Target 

Node 

Scenario Description 

99 Type of alert 

7 Frequency 

3 Public awareness of WEA 

20 History of relevance 


< 

CQ 

o 

Q 

LU 

u. 

C3 

X 





The alert pertains to an imminent threat (i.e., a flash flood) in 
your area 
















Case 3: imminent 
threat, many prior 
alerts, sufficient public 

View as 

Over the oast 6 months, vou have received manv alerts - 48 
from the National Weather Service regarding severe weather 
events, 12 AMBER alerts, and 18 alerts from your city’s EMA 
regarding local threats. 

1 

0 

1 

0 


3 

6 

3 

1 

5 

4 

6 

6 

2 


outreach, history of 
irrelevance 

Spam 

You have oreviouslv been made aware of the WEA orooram 
via public outreach (e.g., newspaper articles, mobile service 
provider mailings, TV and radio news stories) from the EMA. 















3 



You believe that most of the prior alerts that you have re¬ 
ceived were not relevant to you. 

















The alert pertains to an imminent threat (i.e., a flash flood) in 
your area 
















Case 4: imminent 
threat, many prior 
alerts, insufficient pub- 

View as 

Over the oast 6 months, vou have received manv alerts - 48 
from the National Weather Service regarding severe weather 
events, 12 AMBER alerts, and 18 alerts from your city’s EMA 
regarding local threats. 

1 

0 

0 

1 


3 

3 

4 

1 

4 

3 

4 

2 

2 


lie outreach, history of 
relevance 

Spam 

You have not been made aware of the WEA orooram via 
public outreach (e.g., newspaper articles, mobile service 
provider mailings, TV and radio news stories) from the EMA. 


















You believe that most of the prior alerts that you have re¬ 
ceived were relevant to you. 
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Scenarios 

Factors 

Responses 

# 

Case 

Target 

Node 

Scenario Description 

99 Type of alert 

7 Frequency 

3 Public awareness of WEA 

20 History of relevance 


< 

CQ 

o 

Q 

LU 

u. 

C3 

X 





The alert is an AMBER alert oertainino to a child abduction in 
your area 
















Case 5: AMBER alert, 
few alerts prior, suffi¬ 
cient public outreach, 
history of irrelevance 

View as 

Over the oast 6 months, vou have received few alerts - 6 from 
the National Weather Service regarding severe weather 
events, 1 AMBER alert, and 2 alerts from your city’s EMA 
regarding local threats. 

0 

1 

1 

0 


2 

5 

4 

1 

3 

4 

5 

4 

2 


Spam 

You have oreviouslv been made aware of the WEA oroaram 
via public outreach (e.g., newspaper articles, mobile service 
provider mailings, TV and radio news stories) from the EMA. 















3 



You believe that most of the prior alerts that you have re¬ 
ceived were not relevant to you. 

















The alert is an AMBER alert oertainino to a child abduction in 
your area 
















Case 6: AMBER alert, 
few prior alerts, insuffi¬ 
cient public outreach, 
history of relevance 

View as 

Over the oast 6 months, vou have received few alerts - 6 from 
the National Weather Service regarding severe weather 
events, 1 AMBER alert, and 2 alerts from your city’s EMA 
regarding local threats. 

0 

1 

0 

1 


3 

6 

4 

1 

4 

3 

2 

2 

2 


Spam 

You have not been made aware of the WEA oroaram via 
public outreach (e.g., newspaper articles, mobile service 
provider mailings, TV and radio news stories) from the EMA. 


















You believe that most of the prior alerts that you have re¬ 
ceived were relevant to you. 
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Scenarios 

Factors 

Responses 

# 

Case 

Target 

Node 

Scenario Description 

99 Type of alert 

7 Frequency 

3 Public awareness of WEA 

20 History of relevance 


< 

CQ 

o 

Q 

LU 

u. 

C3 

X 





The alert is an AMBER alert oertainino to a child abduction in 
your area 
















Case 7: AMBER alert, 
many prior alerts, suf¬ 
ficient public outreach, 
history of relevance 

View as 

Over the oast 6 months, vou have received many alerts - 48 
from the National Weather Service regarding severe weather 
events, 12 AMBER alerts, and 18 alerts from your city’s EMA 
regarding local threats. 

0 

0 

1 

1 


2 

2 

5 

1 

3 

1 

3 

2 

2 


Spam 

You have oreviouslv been made aware of the WEA oroaram 
via public outreach (e.g., newspaper articles, mobile service 
provider mailings, TV and radio news stories) from the EMA. 















3 



You believe that most of the prior alerts that you have re¬ 
ceived were relevant to you. 

















The alert is an AMBER alert oertainino to a child abduction in 
your area 
















Case 8: AMBER alert, 
many prior alerts, in¬ 
sufficient public out- 

View as 

Over the oast 6 months, vou have received many alerts - 48 
from the National Weather Service regarding severe weather 
events, 12 AMBER alerts, and 18 alerts from your city’s EMA 
regarding local threats. 

0 

0 

0 

0 


4 

5 

3 

1 

4 

5 

6 

2 

2 


reach, history of 
irrelevance 

Spam 

You have not been made aware of the WEA oroaram via 
public outreach (e.g., newspaper articles, mobile service 
provider mailings, TV and radio news stories) from the EMA. 


















You believe that most of the prior alerts that you have re¬ 
ceived were not relevant to you. 
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Scenarios 

Factors 

Responses 

# 

Case 

Target 

Node 

Scenario Description 

1 Relevance 

21 Clarity of message spelling and grammar 

44 Redundancy of alerting 



< 

CQ 

o 

Q 

LU 

u. 

O 

X 



Case 1: relevant, 
clear, unconfirmed, 

View as 
spam 

You determine that the alert is relevant to vou (e.a., it aoDlies 
to your location, it is issued in a timely manner, and it ad¬ 
dresses an emergency that affects you). 
















The message is clear and understandable with no soelling or 
grammar errors 

1 

1 

0 



4 

5 

5 

2 

5 

2 

4 

4 

4 

4 



You can find no confirmation of the WEA alert information via 
other channels such as radio or TV news. 















Case 2: relevant, un¬ 
clear, confirmed 

View as 
spam 

You determine that the alert is relevant to vou (e.g., it aoblies 
to your location, it is issued in a timely manner, and it ad¬ 
dresses an emergency that affects you). 
















The message is not clear and contains some soelling and 
grammar errors 

1 

0 

1 



3 

3 

4 

1 

4 

2 

3 

4 

3 




You can confirm WEA alert information via other channels 
such as radio or TV news. 
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Scenarios 

Factors 

Responses 

# 

Case 

Target 

Node 

Scenario Description 

1 Relevance 

21 Clarity of message spelling and grammar 

44 Redundancy of alerting 



< 

CQ 

O 

Q 

LU 

u. 

o 

X 



Case 3: irrelevant, 

View as 

You determine that the alert is not relevant to vou (e.a., it 
does not apply to your location, it is not issued in a timely 
manner, or it does not address an emergency that affects 
you). 

0 

1 

1 






1 





1 


clear, confirmed 

Spam 

The message is clear and understandable with no soellino or 
grammar errors 



2 

2 

3 

2 

6 

2 

3 

4 



You can confirm WEA alert information via other channels 
such as radio or TV news. 















Case 4: irrelevant. 

View as 

You determine that the alert is not relevant to vou (e.a.. it 
does not apply to your location, it is not issued in a timely 
manner, or it does not address an emergency that affects 
you). 

0 

0 

0 













unclear, unconfirmed 

Spam 

The message is not clear and contains some soelling and 
grammar errors 



7 

7 

6 

4 

7 

7 

7 

6 

5 




You can find no confirmation of the WEA alert information via 
other channels such as radio or TV news. 
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Scenarios 

Factors 

Responses 

# 

Case 

Target 

Node 

Scenario Description 

coordinated 

44 Redundancy of alerting 

48 Alerts viewed as spam 



< 

CQ 

u 

Q 

LU 

u. 

CD 

X 


5 

Case 1: coordinated 
alerts, confirmed, con¬ 
sidered spam 

Opt out 

The two alerts aooear to be coordinated, or contain the same 
information. 

1 

1 

0 



5 

3 

4 

2 

4 

2 

3 

6 

2 

You can confirm alert information via other channels such as 
radio or TV news. 

You believe that the WEA alerts are soam. 

Case 2: coordinated 
alerts, unconfirmed, 
not considered spam 

Opt out 

The two alerts aobear to be coordinated, or contain the same 
information. 

1 

0 

1 



3 

2 

3 

3 

3 

1 

3 

2 

2 

You can find no confirmation of the alert information via other 
channels such as radio or TV news. 

You believe that the WEA alerts are not soam. 

Case 3: uncoordinated 
alerts, confirmed, not 
considered spam 

Opt out 

The two alerts are uncoordinated, and do not aaree. 

0 

1 

1 



3 

2 

4 

1 

3 

1 

3 

2 

1 

You can confirm alert information via other channels such as 
radio or TV news. 

You believe that the WEA alerts are not soam. 

Case 4: uncoordinated 
alerts, unconfirmed, 
considered spam 

Opt out 

The two alerts are uncoordinated, and do not aaree. 

0 

0 

0 



5 

4 

5 

3 

3 

6 

6 

6 

4 

You can find no confirmation of the alert information via other 
channels such as radio or TV news. 

You believe that the WEA alerts are soam. 
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Scenarios 

Factors 

Responses 

# 

Case 

Target 

Node 

Scenario Description 

55 Local jurisdictions act uncoordinated 

21 Clarity of message spelling and grammar 

71 Message in primary language 

15 Easy add’l follow-us mechanisms 


< 


u 

Q 

LU 

u. 

CD 

X 



Case 1: coordinated 
alerts, clear message, 
in primary language, 
‘‘follow us” mecha- 


The two alerts aooear to be coordinated, or contain the same 
information. 
















Understand 

The messages are clear and understandable with no soelling 
or grammar errors 

1 

1 

1 

1 


7 

7 

6 

7 

7 

7 

7 

7 

7 


The messages are written in your orimarv language 



nisms present 


The EMAs have alternate communications channels (e.g., 
web site, Facebook page, Twitter account) that you can ac¬ 
cess easily. 















6 



The two alerts aobear to be coordinated, or contain the same 
information. 
















Case 2: coordinated 
alerts, clear message. 


The messages are clear and understandable with no soelling 
or grammar errors 
















not in primary lan¬ 
guage, “follow us” 

Understand 

The messages are issued in a language that you understand, 
but one that is not vour orimarv language. 

1 

1 

0 

0 


5 

7 

4 

7 

5 

6 

6 

6 

7 


mechanisms absent 


The EMAs have no alternate communications channels (e.g., 
web site, Facebook page, Twitter account) that you can ac¬ 
cess easily. 
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Scenarios 

Factors 

Responses 

# 

Case 

Target 

Node 

Scenario Description 

55 Local jurisdictions act uncoordinated 

21 Clarity of message spelling and grammar 

71 Message in primary language 

15 Easy add’l follow-us mechanisms 


< 


u 

Q 

LU 

u. 

CD 

X 



Case 3: coordinated 
alerts, unclear mes¬ 
sage, in primary lan¬ 
guage, “follow us” 
mechanisms absent 


The two alerts aooear to be coordinated, or contain the same 
information. 
















Understand 

The messages are not clear and contain some soelling and 
grammar errors 

1 

0 

1 

0 


5 

6 

4 

4 

5 

5 

5 

4 

6 


The messages are written in your orimarv language 




The EMAs have no alternate communications channels (e.g., 
web site, Facebook page, Twitter account) that you can ac¬ 
cess easily. 















6 



The two alerts aobear to be coordinated, or contain the same 
information. 
















Case 4: coordinated 
alerts, unclear mes- 


The messages are not clear and contain some soelling and 
grammar errors 
















sage, not in primary 
language, “follow us” 

Understand 

The messages are issued in a language that you understand, 
but one that is not vour orimarv language. 

1 

0 

0 

1 


4 

6 

4 

5 

4 

5 

4 

4 

4 


mechanisms present 


The EMAs have alternate communications channels (e.g., 
web site, Facebook page, Twitter account) that you can ac¬ 
cess easily. 
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Scenarios 

Factors 

Responses 

# 

Case 

Target 

Node 

Scenario Description 

55 Local jurisdictions act uncoordinated 

21 Clarity of message spelling and grammar 

71 Message in primary language 

15 Easy add’l follow-us mechanisms 


< 


u 

Q 

LU 

u. 

CD 

X 





The two alerts are uncoordinated, and do not aaree. 
















Case 5: uncoordinated 
alerts, clear message, 


The messages are clear and understandable with no soelling 
or grammar errors 
















in primary language. 

Understand 

The messages are written in your orimarv language 

0 

1 

1 

0 


7 

6 

3 

4 

5 

4 

4 

4 

2 


“follow us” mecha¬ 
nisms absent 


The EMAs have no alternate communications channels (e.g., 
web site, Facebook page, Twitter account) that you can ac¬ 
cess easily. 















6 



The two alerts are uncoordinated, and do not agree. 
















Case 6: uncoordinated 
alerts, clear message, 
not in primary lan¬ 
guage, “follow us” 
mechanisms present 


The messages are clear and understandable with no soelling 
or grammar errors 
















Understand 

The messages are issued in a language that you understand, 
but one that is not vour orimarv language. 

0 

1 

0 

1 


6 

6 

4 

4 

5 

4 

4 

4 

4 



The EMAs have alternate communications channels (e.g., 
web site, Facebook page, Twitter account) that you can ac¬ 
cess easily. 
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Scenarios 


Factors 


Responses 


Case 


Target 

Node 


Scenario Description 


•o 

c 

O) 

c 

o 

a 

(/) 

0) 

O) 

(Q 

(/) 

(/) 

0) 

E 

o 

>. 


o 


■D 

■D 

ra 

>. 

U) 

ra 

liJ 


CQ 


O 


CD 


The two alerts are uncoordinated, and do not agree. 


Case 7: uncoordinated 
alerts, unclear mes¬ 
sage, in primary lan¬ 
guage, “follow us” 
mechanisms present 


The messages are not clear and contain some spelling and 
grammar errors 


Understand 


The messages are written in your primary language 


The EMAs have alternate communications channels (e.g., 
web site, Facebook page, Twitter account) that you can ac¬ 
cess easily. 


The two alerts are uncoordinated, and do not agree. 


Case 8: uncoordinated 
alerts, unclear mes¬ 
sage, not in primary 
language, “follow us” 
mechanisms absent 


The messages are not clear and contain some spelling and 
grammar errors 


Understand 


The messages are issued in a language that you understand, 
but one that is not your primary language. 


The EMA has no alternate communications channels (e.g., 
web site, Facebook page, Twitter account) that you can ac¬ 
cess easily. 
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Scenarios 

Factors 

Responses 

# 

Case 

Target 

Node 

Scenario Description 

30 Explain what has happened 

70 Explain why I should act 

1 Relevance 

21 Clarity of message spelling and grammar 

37 Confirmation via social media 

< 


u 

Q 

LU 

u. 

CD 

X 


7 

Case 1: includes 
"what,” includes "why,” 
relevant, clear mes¬ 
sage, not confirmed 
via social media 

Believe 

"Nuclear plant radiation release 
in this area 

Take shelter now 

to avoid radiation exposure” 

1 

1 

1 

1 

0 

5 

6 

4 

7 

6 

6 

5 

6 

6 

You determine that the alert is relevant to vou (e.a., it aoDlies 
to your location, it is issued in a timely manner, and it ad¬ 
dresses an emergency that affects you). 

You cannot find confirmation of the information on social 
media (e.g., Twitter, Facebook) 

Case 2: includes 
"what,” includes "why,” 
irrelevant, unclear 
message, confirmed 
via social media 

Believe 

"Nucular plant radiatin release 
in this area 

Take shelter now 
to avoid radiatin expossur” 

1 

1 

0 

0 

1 

5 

3 

3 

7 

5 

6 

4 

3 

2 

You determine that the alert is not relevant to vou (e.a., it 
does not apply to your location, it is not issued in a timely 
manner, or it does not address an emergency that affects 
you). 

You can find confirmation of the information on social media 
(e.g., Twitter, Facebook) 
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Scenarios 

Factors 

Responses 

# 

Case 

Target 

Node 

Scenario Description 

30 Explain what has happened 

70 Explain why I should act 

1 Relevance 

21 Clarity of message spelling and grammar 

37 Confirmation via social media 

< 


u 

Q 

LU 

u. 

CD 

X 



Case 3: includes 


“Nucular plant radiatin release 
in this area 

Take shelter now” 
















“what,” omits “why,” 
relevant, unclear mes¬ 
sage, confirmed via 
social media 

Believe 

You determine that the alert is relevant to vou (e.a., it aoDlies 
to your location, it is issued in a timely manner, and it ad¬ 
dresses an emergency that affects you). 

1 

0 

1 

0 

1 

4 

5 

5 

5 

6 

6 

4 

5 

3 

7 


You can find confirmation of the information on social media 
(e.g., Twitter, Facebook) 


















“Nuclear plant radiation release in this area Take shelter now” 
















Case 4: includes 
“what,” omits “why,” 
irrelevant, clear mes¬ 
sage, unconfirmed via 

Believe 

You determine that the alert is not relevant to vou (e.a., it 
does not apply to your location, it is not issued in a timely 
manner, or it does not address an emergency that affects 
you). 

1 

0 

0 

1 

0 

4 

4 

4 

6 

4 

5 

3 

4 

5 


social media 


You cannot find confirmation of the information on social 
media (e.g., Twitter, Facebook) 
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Scenarios 

Factors 

Responses 

# 

Case 

Target 

Node 

Scenario Description 

30 Explain what has happened 

70 Explain why I should act 

1 Relevance 

21 Clarity of message spelling and grammar 

37 Confirmation via social media 

< 


u 

Q 

LU 

u. 

O 

X 



Case 5: omits “what,” 
includes “why,” rele¬ 
vant, unclear mes¬ 
sage, unconfirmed via 
social media 


“Nucular power plant warnign 
in this area 

Take shelter now 
to avoid radiatin expossur” 
















Believe 

You determine that the alert is relevant to vou (e.a., it aoDlies 
to your location, it is issued in a timely manner, and it ad¬ 
dresses an emergency that affects you). 

0 

1 

1 

0 

0 

3 

3 

4 

6 

4 

5 

2 

3 

3 




You cannot find confirmation of the information on social 
media (e.g., Twitter, Facebook) 















7 

Case 6: omits “what,” 


“Nuclear Power plant warning 
in this area 

Take shelter now 

to avoid radiation exposure” 
















includes “why,” irrele¬ 
vant, clear message, 
confirmed via social 
media 

Believe 

You determine that the alert is not relevant to vou (e.a., it 
does not apply to your location, it is not issued in a timely 
manner, or it does not address an emergency that affects 
you). 

0 

1 

0 

1 

1 

4 

6 

5 

7 

7 

6 

7 

5 

6 




You can find confirmation of the information on social media 
(e.g., Twitter, Facebook) 
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Scenarios 

Factors 

Responses 

# 

Case 

Target 

Node 

Scenario Description 

30 Explain what has happened 

70 Explain why I should act 

1 Relevance 

21 Clarity of message spelling and grammar 

37 Confirmation via social media 

< 


u 

Q 

LU 

u. 

CD 

X 



Case 7: omits “what,” 


“Nuclear Power plant warning 
in this area 

Take shelter now” 
















omits “why,” relevant, 
clear message, con¬ 
firmed via social media 

Believe 

You determine that the alert is relevant to vou (e.a., it aoDlies 
to your location, it is issued in a timely manner, and it ad¬ 
dresses an emergency that affects you). 

0 

0 

1 

1 

1 

4 

5 

5 

7 

7 

7 

7 

5 

6 




You can find confirmation of the information on social media 
(e.g., Twitter, Facebook) 















7 

Case 8: omits “what,” 


“Nucular Power plant warnign 
in this area 

Take shelter now” 
















omits “why,” irrelevant, 
unclear message, 
unconfirmed via social 
media 

Believe 

You determine that the alert is not relevant to vou (e.a., it 
does not apply to your location, it is not issued in a timely 
manner, or it does not address an emergency that affects 
you). 

0 

0 

0 

0 

0 

2 

2 

3 

4 

1 

2 

2 

3 

2 




You cannot find confirmation of the information on social 
media (e.g., Twitter, Facebook) 
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Scenarios 

Factors 

Responses 

# 

Case 

Target 

Node 

Scenario Description 

44 Redundancy of alerting 

20 History of relevance 

55 Local jurisdictions act uncoordinated 

3 Public awareness of WEA 


< 

CQ 

o 

Q 

LU 

u. 

C3 

X 





You can confirm alert information via other channels such as 
radio or TV news. 
















Case 1: confirmed, 
history of relevance, 
coordinated, public 
outreach 


You believe that most of the prior alerts that you have re¬ 
ceived were relevant to you. 
















Believe 

The two alerts aobear to be coordinated, or contain the same 
information. 

1 

1 

1 

1 


7 

6 

6 

7 

7 

7 

7 

6 

7 

8 


You have oreviouslv been made aware of the WEA oroaram 
via public outreach (e.g., newspaper articles, mobile service 
provider mailings, TV and radio news stories) from the EMA. 


















You can confirm alert information via other channels such as 
radio or TV news. 
















Case 2: confirmed, 
history of relevance. 

Believe 

You believe that most of the prior alerts that you have re¬ 
ceived were relevant to you. 

1 

1 

0 

0 












uncoordinated, no 

The two alerts are uncoordinated, and do not aaree. 


4 

5 

4 

6 

5 

6 

4 

5 

5 


public outreach 


You have not been made aware of the WEA oroaram via 
public outreach (e.g., newspaper articles, mobile service 
provider mailings, TV and radio news stories) from the EMAs. 
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Scenarios 

Factors 

Responses 

# 

Case 

Target 

Node 

Scenario Description 

44 Redundancy of alerting 

20 History of relevance 

55 Local jurisdictions act uncoordinated 

3 Public awareness of WEA 


< 

CQ 

o 

Q 

LU 

u. 

C3 

X 





You can confirm alert information via other channels such as 
radio or TV news. 
















Case 3: confirmed, 
history of irrelevance, 
coordinated, no public 
outreach 


You believe that most of the prior alerts that you have re¬ 
ceived were not relevant to you. 
















Believe 

The two alerts aobear to be coordinated, or contain the same 
information. 

1 

0 

1 

0 


4 

6 

4 

7 

5 

3 

6 

3 

5 

8 


You have not been made aware of the WEA oroaram via 
public outreach (e.g., newspaper articles, mobile service 
provider mailings, TV and radio news stories) from the EMAs. 


















You can confirm alert information via other channels such as 
radio or TV news. 
















Case 4: confirmed, 
history of irrelevance. 

Believe 

You believe that most of the prior alerts that you have re¬ 
ceived were not relevant to you. 

1 

0 

0 

1 












uncoordinated, public 

The two alerts are uncoordinated, and do not aaree. 


5 

4 

3 

4 

4 

3 

5 

3 

6 


outreach 


You have oreviouslv been made aware of the WEA oroaram 
via public outreach (e.g., newspaper articles, mobile service 
provider mailings, TV and radio news stories) from the EMAs. 
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Scenarios 

Factors 

Responses 

# 

Case 

Target 

Node 

Scenario Description 

44 Redundancy of alerting 

20 History of relevance 

55 Local jurisdictions act uncoordinated 

3 Public awareness of WEA 


< 

CQ 

o 

Q 

LU 

u. 

C3 

X 





You can find no confirmation of the alert information via other 
channels such as radio or TV news. 
















Case 5: unconfirmed, 
history of relevance, 
coordinated, no public 
outreach 


You believe that most of the prior alerts that you have re¬ 
ceived were relevant to you. 
















Believe 

The two alerts aobear to be coordinated, or contain the same 
information. 

0 

1 

1 

0 


5 

6 

4 

7 

5 

5 

4 

5 

6 

8 


You have not been made aware of the WEA oroaram via 
public outreach (e.g., newspaper articles, mobile service 
provider mailings, TV and radio news stories) from the EMAs. 


















You can find no confirmation of the alert information via other 
channels such as radio or TV news. 
















Case 6: unconfirmed, 
history of relevance. 

Believe 

You believe that most of the prior alerts that you have re¬ 
ceived were relevant to you. 

0 

1 

0 

1 












uncoordinated, public 

The two alerts are uncoordinated, and do not aaree. 


4 

5 

4 

5 

5 

5 

3 

4 

5 


outreach 


You have oreviouslv been made aware of the WEA oroaram 
via public outreach (e.g., newspaper articles, mobile service 
provider mailings, TV and radio news stories) from the EMAs. 
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Scenarios 

Factors 

Responses 

# 

Case 

Target 

Node 

Scenario Description 

44 Redundancy of alerting 

20 History of relevance 

55 Local jurisdictions act uncoordinated 

3 Public awareness of WEA 


< 

CQ 

o 

Q 

LU 

u. 

C3 

X 





You can find no confirmation of the alert information via other 
channels such as radio or TV news. 
















Case 7: unconfirmed, 
history of irrelevance, 
coordinated, public 
outreach 


You believe that most of the prior alerts that you have re¬ 
ceived were not relevant to you. 
















Believe 

The two alerts aobear to be coordinated, or contain the same 
information. 

0 

0 

1 

1 


3 

4 

5 

6 

4 

3 

4 

5 

6 

8 


You have oreviouslv been made aware of the WEA oroaram 
via public outreach (e.g., newspaper articles, mobile service 
provider mailings, TV and radio news stories) from the EMAs. 


















You can find no confirmation of the alert information via other 
channels such as radio or TV news. 
















Case 8: unconfirmed, 
history of irrelevance. 

Believe 

You believe that most of the prior alerts that you have re¬ 
ceived were not relevant to you. 

0 

0 

0 

0 


1 




2 

2 

1 

2 



uncoordinated, no 

The two alerts are uncoordinated, and do not aaree. 


2 

3 

3 

4 


public outreach 


You have not been made aware of the WEA oroaram via 
public outreach (e.g., newspaper articles, mobile service 
provider mailings, TV and radio news stories) from the EMAs. 
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Scenarios 


Factors 


Responses 


Case 


Target 

Node 


Scenario Description 


m 


o 


CD 


Prior WEA alerts that you have received have included an “all 
clear” message after the event has been resolved? 


Case 1: final commu¬ 
nication, local alert 
source, few prior 
alerts, references for 
more info 


The alert addresses an imminent threat to your area and is 
sent by your local Emergency Management Agency 


Believe 


Over the past 6 months, you have received few alerts - 6 from 
the National Weather Service regarding severe weather 
events, 1 AMBER alert, and 2 alerts from your city’s EMA 
regarding local threats. 


The message includes a recommendation to ‘‘Check TV & 
Radio for more info” 


9 


Case 2: final commu¬ 
nication, local alert 
source, many prior 
alerts, no references 
for more info 


Believe 


Prior WEA alerts that you have received have included an “all 
clear” message after the event has been resolved? 

The alert addresses an imminent threat to your area and is 
sent by your local Emergency Management Agency 

Over the past 6 months, you have received many alerts - 48 
from the National Weather Service regarding severe weather 
events, 12 AMBER alerts, and 18 alerts from your city’s EMA 
regarding local threats. 

The message does not include a recommendation to ‘‘Check 
TV & Radio for more info” 
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Scenarios 


Factors 


Responses 


Case 


Target 

Node 


Scenario Description 


m 


u 


o 


Prior WEA alerts that you have received have included an “all 
clear” message after the event has been resolved? 


Case 3: final commu¬ 
nication, national alert 
source, few prior 
alerts, no references 
for more info 


The alert addresses an imminent threat to your area and is 
sent by a national government agency (e.g., FBI, FEMA). 


Believe 


Over the past 6 months, you have received few alerts - 6 from 
the National Weather Service regarding severe weather 
events, 1 AMBER alert, and 2 alerts from your city’s EMA 
regarding local threats. 


The message does not include a recommendation to ‘‘Check 
TV & Radio for more info” 


Case 4: final commu¬ 
nication, national alert 
source, many prior 
alerts, references for 
more info 


Believe 


Prior WEA alerts that you have received have included an “all 
clear” message after the event has been resolved? 

The alert addresses an imminent threat to your area and is 
sent by a national government agency (e.g., FBI, FEMA). 

Over the past 6 months, you have received many alerts - 48 
from the National Weather Service regarding severe weather 
events, 12 AMBER alerts, and 18 alerts from your city’s EMA 
regarding local threats. 

The message includes a recommendation to “Check TV & 
Radio for more info” 
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Scenarios 


Factors 


Responses 


Case 


Target 

Node 


Scenario Description 


m 


u 


o 


Prior WEA alerts that you have received have not included an 
"all clear” message after the event has been resolved? 


Case 5: no final com¬ 
munication, local alert 
source, few prior 
alerts, no references 
for more info 


The alert addresses an imminent threat to your area and is 
sent by your local Emergency Management Agency 


Believe 


Over the past 6 months, you have received few alerts - 6 from 
the National Weather Service regarding severe weather 
events, 1 AMBER alert, and 2 alerts from your city’s EMA 
regarding local threats. 


The message does not include a recommendation to "Check 
TV & Radio for more info” 


Case 6: no final com¬ 
munication, local alert 
source, many prior 
alerts, references for 
more info 


Believe 


Prior WEA alerts that you have received have not included an 
"all clear” message after the event has been resolved? 

The alert addresses an imminent threat to your area and is 
sent by your local Emergency Management Agency 

Over the past 6 months, you have received many alerts - 48 
from the National Weather Service regarding severe weather 
events, 12 AMBER alerts, and 18 alerts from your city’s EMA 
regarding local threats. 

The message includes a recommendation to "Check TV & 
Radio for more info” 


0 
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Scenarios 


Factors 


Responses 


Case 


Target 

Node 


Scenario Description 


m 


u 


o 


Prior WEA alerts that you have received have not included an 
"all clear” message after the event has been resolved? 


Case 7: no final com¬ 
munication, national 
alert source, few prior 
alerts, references for 
more info 


The alert addresses an imminent threat to your area and is 
sent by a national government agency (e.g., FBI, FEMA). 


Believe 


Over the past 6 months, you have received few alerts - 6 from 
the National Weather Service regarding severe weather 
events, 1 AMBER alert, and 2 alerts from your city’s EMA 
regarding local threats. 


The message includes a recommendation to "Check TV & 
Radio for more info” 


Prior WEA alerts that you have received have not included an 
"all clear” message after the event has been resolved? 


Case 8: no final com¬ 
munication, national 
alert source, many 
prior alerts, no refer¬ 
ences for more info 


The alert addresses an imminent threat to your area and is 
sent by a national government agency (e.g., FBI, FEMA). 


Believe 


Over the past 6 months, you have received many alerts - 48 
from the National Weather Service regarding severe weather 
events, 12 AMBER alerts, and 18 alerts from your city’s EMA 
regarding local threats. 


The message does not include a recommendation to "Check 
TV & Radio for more info” 


CMU/SEI-2013-SR-021 | 238 





































Appendix V Public Trust Model Validation Survey 


TRUST MODEL VALIDATION SCENARIOS 

Scenario 1. This set of scenarios probes the impact of an alert message’s specification of what 

has happened, who should act, and why they should act on your interpretation of the 
alert. 

Your city’s Emergency Management Agency (EMA) has been using WEA to alert the public to 
imminent threats for the past 12 months. From public service announcements on the TV news and 
in the newspapers, you are aware of what the WEA service is and how it is used. You have re¬ 
ceived several WEA alerts on you mobile phone over the past year. 

For each of the cases below, please indicate how likely you are to consider the received alert as 
being relevant to you. 


Case 1: specifies “what has happened”, omits “who should act”, specifies “why they should 

_ act” _ 

You are in ZIP code 12345 at 4:37 PM, when your mobile phone receives and displays the follow¬ 
ing alert 

“Nuclearplant radiation release 

in this area 

Take shelter now 

to avoid radiation exposure ’’ 

How likely are you to consider this alert to be relevant to you? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 
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Case 2: specifies “what has happened”, specifies “who should act”, omits “why they should 

_ act” _ 

You are in ZIP code 12345 at 4:37 PM, when your mobile phone receives and displays the follow¬ 
ing alert 

“Nuclearplant radiation release 
in ZIP 12345 
Take shelter now” 


How likely are you to consider this alert to be relevant to you? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 


Case 3: omits “what has happened”, specifies “who should act”, specifies “why they should 

_ act” _ 

You are in ZIP code 12345 at 4:37 PM, when your mobile phone receives and displays the follow¬ 
ing alert 

“Nuclear power plant warning 

in ZIP 12345 

Take shelter now 

to avoid radiation exposure" 

How likely are you to consider this alert to be relevant to you? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 


Case 4: omits “what has happened”, omits “who should act”, omits “why they should act” 
You are in ZIP code 12345 at 4:37 PM, when your mobile phone receives and displays the follow¬ 
ing alert 

“Nuclear power plant warning 

in this area 

Take shelter now" 

How likely are you to consider this alert to be relevant to you? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 
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Scenario 2. This set of scenarios probes the impact of an alert’s specification of action to take, 
time to act, who should act, the lead time for the alert, and the relevance of the alert 
on your willingness to take action. 

Your city’s Emergency Management Agency (EMA) has been using WEA to alert the public to 
imminent threats for the past 12 months. You have received WEA alerts on you mobile phone 
over the past year. 

For each of the cases below, please indicate how likely you are to take action in response to the 
alert. 


Case 1: defined action, defined time, sufficient lead, includes “who, ” irrelevant _ 

You are in ZIP code 12345 at 4:37 PM, when your mobile phone receives and displays the follow¬ 
ing alert 
“Flash flood 
in ZIP 12345 
from 4:55 to 5:35 PM 
Evacuate low lying areas ” 

You determine that the alert is not relevant to you (e.g., it does not apply to your location, it is not 
issued in a timely manner, it does not address an emergency that affects you). 

How likely are you to take action in response to this alert? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 


Case 2: defined action, defined time, insufficient lead, omits “who, ” relevant _ 

You are in ZIP code 12345 at 4:37 PM, when your mobile phone receives and displays the follow¬ 
ing alert 
“Flash flood 
in this area 

from 4:40 to 5:20 PM 
Evacuate low lying areas ” 

You determine that the alert is relevant to you (e.g., it applies to your location, it is issued in a 
timely manner, it addresses an emergency that affects you). 

How likely are you to take action in response to this alert? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 
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Case 3: defined action, undefined time, sufficient lead, omits “who, ” relevant _ 

You are in ZIP code 12345 at 4:37 PM, when your mobile phone receives and displays the follow¬ 
ing alert 
“Flash flood 
in this area 
at 4:55 PM 

Evacuate low lying areas ” 

You determine that the alert is relevant to you (e.g., it applies to your location, it is issued in a 
timely manner, it addresses an emergency that affects you). 

How likely are you to take action in response to this alert? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 


Case 4: defined action, undefined time, insufficient lead, includes “who, ” irrelevant _ 

You are in ZIP code 12345 at 4:37 PM, when your mobile phone receives and displays the follow¬ 
ing alert 
“Flash flood 
in ZIP 12345 
at 4:40 PM 

Evacuate low lying areas ” 

You determine that the alert is not relevant to you (e.g., it does not apply to your location, it is not 
issued in a timely manner, it does not address an emergency that affects you). 

How likely are you to take action in response to this alert? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 
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Case 5: undefined action, defined time, sufficient lead, omits “who, ” irrelevant _ 

You are in ZIP code 12345 at 4:37 PM, when your mobile phone receives and displays the follow¬ 
ing alert 
“Flash flood 
in this area 

from 4:55 to 5:35 PM" 

You determine that the alert is not relevant to you (e.g., it does not apply to your location, it is not 
issued in a timely manner, it does not address an emergency that affects you). 

How likely are you to take action in response to this alert? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 


Case 6: undefined action, defined time, insufficient lead, includes “who, ” relevant _ 

You are in ZIP code 12345 at 4:37 PM, when your mobile phone receives and displays the follow¬ 
ing alert 
“Flash flood 
in ZIP 12345 
from 4:40 to 5:20 PM" 

You determine that the alert is relevant to you (e.g., it applies to your location, it is issued in a 
timely manner, it addresses an emergency that affects you). 

How likely are you to take action in response to this alert? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 
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Case 7: undefined action, undefined time, sufficient lead, includes “who, ” relevant _ 

You are in ZIP code 12345 at 4:37 PM, when your mobile phone receives and displays the follow¬ 
ing alert 
“Flash flood 
in ZIP 12345 
from 4:55 PM” 

You determine that the alert is relevant to you (e.g., it applies to your location, it is issued in a 
timely manner, it addresses an emergency that affects you). 

How likely are you to take action in response to this alert? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 


Case 8: undefined action, undefined time, insufficient lead, omits “who, ” irrelevant _ 

You are in ZIP code 12345 at 4:37 PM, when your mobile phone receives and displays the follow¬ 
ing alert 
“Flash flood 
in this area 
at 4:40 PM 

You determine that the alert is not relevant to you (e.g., it does not apply to your location, it is not 
issued in a timely manner, it does not address an emergency that affects you). 

How likely are you to take action in response to this alert? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 
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Scenario 3. This set of scenarios probes the impact of the type of alert, the frequency of alerts, 

public outreach, and the history of alert relevance on you interpretation of the alert 
as spam 

Your city’s Emergency Management Agency (EMA) has been using WEA to alert the public to 
imminent threats for the past 12 months. You have received WEA alerts on you mobile phone 
over the past year. 

At 4:37 PM your mobile phone receives and displays an alert. 

For each of the cases below, please indicate how likely you are to view this alert as spam 


Case 1: imminent threat, few prior alerts, sufficient public outreach, history of relevance _ 

The alert pertains to an imminent threat (i.e., a flash flood) in your area 

Over the past 6 months, you have received few alerts - 6 from the National Weather Service re¬ 
garding severe weather events, and 1 AMBER alert, and 2 alerts from your city’s EMA regarding 
local threats. 

You have previously been made aware of the WEA program via public outreach (e.g., newspaper 
articles, mobile service provider mailings, TV and radio news stories) from the EMA. 

You believe that most of the prior alerts that you have received were relevant to you. 

How likely you are to view this alert as spam? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 


Case 2: imminent threat, few prior alerts, insufficient public outreach, history of irrelevance 
The alert pertains to an imminent threat (i.e., a flash flood) in your area 

Over the past 6 months, you have received few alerts - 6 from the National Weather Service re¬ 
garding severe weather events, and 1 AMBER alert, and 2 alerts from your city’s EMA regarding 
local threats. 

You have not been made aware of the WEA program via public outreach (e.g., newspaper articles, 
mobile service provider mailings, TV and radio news stories) from the EMA. 

You believe that most of the prior alerts that you have received were not relevant to you. 

How likely you are to view this alert as spam? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 
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Case 3: imminent threat, many prior alerts, sufficient public outreach, history of irrelevance 
The alert pertains to an imminent threat (i.e., a flash flood) in your area 

Over the past 6 months, you have received many alerts - 48 from the National Weather Service re¬ 
garding severe weather events, and 12 AMBER alerts, and 18 alerts from your city’s EMA regard¬ 
ing local threats. 

You have previously been made aware of the WEA program via public outreach (e.g., newspaper 
articles, mobile service provider mailings, TV and radio news stories) from the EMA. 

You believe that most of the prior alerts that you have received were not relevant to you. 

How likely you are to view this alert as spam? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 


Case 4: imminent threat, many prior alerts, insufficient public outreach, history of relevance 
The alert pertains to an imminent threat (i.e., a flash flood) in your area 

Over the past 6 months, you have received many alerts - 48 from the National Weather Service re¬ 
garding severe weather events, and 12 AMBER alerts, and 18 alerts from your city’s EMA regard¬ 
ing local threats. 

You have not been made aware of the WEA program via public outreach (e.g., newspaper articles, 
mobile service provider mailings, TV and radio news stories) from the EMA. 

You believe that most of the prior alerts that you have received were relevant to you. 

How likely you are to view this alert as spam? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 
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Case 5: AMBER alert, few alerts prior, sufficient public outreach, history of irrelevance 

The alert is an AMBER alert pertaining to a child abduction in your area 

Over the past 6 months, you have received few alerts - 6 from the National Weather Service re¬ 
garding severe weather events, and 1 AMBER alert, and 2 alerts from your city’s EMA regarding 
local threats. 

You have previously been made aware of the WEA program via public outreach (e.g., newspaper 
articles, mobile service provider mailings, TV and radio news stories) from the EMA. 

You believe that most of the prior alerts that you have received were not relevant to you. 

How likely you are to view this alert as spam? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 


Case 6: AMBER alert, few prior alerts, insufficient public outreach, history of relevance 

The alert is an AMBER alert pertaining to a child abduction in your area 

Over the past 6 months, you have received few alerts - 6 from the National Weather Service re¬ 
garding severe weather events, and 1 AMBER alert, and 2 alerts from your city’s EMA regarding 
local threats. 

You have not been made aware of the WEA program via public outreach (e.g., newspaper articles, 
mobile service provider mailings, TV and radio news stories) from the EMA. 

You believe that most of the prior alerts that you have received were relevant to you. 

How likely you are to view this alert as spam? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 
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Case 7: AMBER alert, many prior alerts, sufficient public outreach, history of relevance 
The alert is an AMBER alert pertaining to a child abduction in your area 

Over the past 6 months, you have received many alerts - 48 from the National Weather Service re¬ 
garding severe weather events, and 12 AMBER alerts, and 18 alerts from your city’s EMA regard¬ 
ing local threats. 

You have previously been made aware of the WEA program via public outreach (e.g., newspaper 
articles, mobile service provider mailings, TV and radio news stories) from the EMA. 

You believe that most of the prior alerts that you have received were relevant to you. 

How likely you are to view this alert as spam? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 


Case 8: AMBER alert, many prior alerts, insufficient public outreach, history of irrelevance 
The alert is an AMBER alert pertaining to a child abduction in your area 

Over the past 6 months, you have received many alerts - 48 from the National Weather Service re¬ 
garding severe weather events, and 12 AMBER alerts, and 18 alerts from your city’s EMA regard¬ 
ing local threats. 

You have not been made aware of the WEA program via public outreach (e.g., newspaper articles, 
mobile service provider mailings, TV and radio news stories) from the EMA. 

You believe that most of the prior alerts that you have received were not relevant to you. 

How likely you are to view this alert as spam? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 
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Scenario 4. This set of scenarios probes the impact of the relevance of the alert, the clarity of the 
message, and the ability to confirm the alert from other sources on your interpreta¬ 
tion of the alert as spam 

Your city’s Emergency Management Agency (EMA) has been using WEA to alert the public to 
imminent threats for the past 12 months. You have received WEA alerts on you mobile phone 
over the past year. 

At 4:37 PM your mobile phone receives and displays an alert. 

For each of the following cases, please indicate how likely you are to view this alert as spam. 


Case 1: relevant, clear, unconfirmed. 

You determine that the alert is relevant to you (e.g., it applies to your location, it is issued in a 
timely manner, it addresses an emergency that affects you). 

The message is clear and understandable with no spelling or grammar errors 

You can find no confirmation of the WEA alert information via other channels such as radio or 
TV news. 

How likely are you to view this alert as spam? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 


Case 2: relevant, unclear, confirmed 

You determine that the alert is relevant to you (e.g., it applies to your location, it is issued in a 
timely manner, it addresses an emergency that affects you). 

The message is not clear and contains some spelling and grammar errors 

You can confirm the WEA alert information via other channels such as radio or TV news. 

How likely are you to view this alert as spam? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 
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Case 3: irrelevant, clear, confirmed 


You determine that the alert is not relevant to you (e.g., it does not apply to your location, it is not 
issued in a timely manner, it does not address an emergency that affects you). 

The message is clear and understandable with no spelling or grammar errors 

You can confirm WEA alert information via other channels such as radio or TV news. 

How likely are you to view this alert as spam? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 


Case 4: irrelevant, unclear, unconfirmed 

You determine that the alert is not relevant to you (e.g., it does not apply to your location, it is not 
issued in a timely manner, it does not address an emergency that affects you). 

The message is not clear and contains some spelling and grammar errors 

You can find no confirmation of the WEA alert information via other channels such as radio or 
TV news. 

How likely are you to view this alert as spam? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 
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Scenario 5. This set of scenarios probes the impact of the coordination of alerts across multiple 

jurisdictions, the ability to confirm the alert from other sources, and your interpreta¬ 
tion of the alert as spam on your willingness to opt out of the WE A service 

Your city’s Emergency Management Agency (EMA) and your county EMA have both been using 
WEA to alert the public to imminent threats for the past 12 months. From public service an¬ 
nouncements on the TV news and in the newspapers, you are aware of what the WEA service is 
and how it is used. You have received several WEA alerts on you mobile phone over the past year. 

At 4:37 PM your mobile phone receives and displays an alert from your city EMA regarding an 
imminent threat (i.e., a flash flood) in your area. At 4:48 your mobile phone receives and displays 
an alert from your county EMA regarding the same emergency. 

For each of the following cases, please indicate how likely you are to opt out of the WEA service. 


Case 1; coordinated alerts, confirmed, considered spam 

The two alerts appear to be coordinated, or contain the same information. 

You can confirm alert information via other channels such as radio or TV news. 
You believe that the WEA alerts are spam. 

How likely are you to opt out of the WEA service? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 


Case 2: coordinated alerts, unconfirmed, not considered spam 

The two alerts appear to be coordinated, or contain the same information. 

You can find no confirmation of the alert information via other channels such as radio or TV 
news. 

You believe that the WEA alerts are not spam. 

How likely are you to opt out of the WEA service? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 
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Case 3: uncoordinated alerts, confirmed, not considered spam 


The two alerts are uncoordinated, and do not agree. 

You can confirm alert information via other channels such as radio or TV news. 
You believe that the WEA alerts are not spam. 

How likely are you to opt out of the WEA service? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 


Case 4: uncoordinated alerts, unconfirmed, considered spam 
The two alerts are uncoordinated, and do not agree. 

You can find no confirmation of the alert information via other channels such as radio or TV 
news. 

You believe that the WEA alerts are spam. 

How likely are you to opt out of the WEA service? (Please choose one) 

□ □ □ □ □ □ □ 


Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

Not 

Not 



Probably 


(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 
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Scenario 6. This set of scenarios probes the impact of the coordination of alerts across multiple 
jurisdictions, the message clarity, the message language, and alternate communica¬ 
tions channels on your ability to understand an alert 

Your city’s Emergency Management Agency (EMA) has been using WEA to alert the public to 
imminent threats for the past 12 months. You have received WEA alerts on you mobile phone 
over the past year. 

At 4:37 PM your mobile phone receives and displays an alert from your city EMA regarding an 
imminent threat (i.e., a flash flood) in your area. At 4:48 your mobile phone receives and displays 
an alert from your county EMA regarding the same emergency. 

For each of the following cases, please indicate how likely you are to understand the alerts. 


Case 1: coordinated alerts, clear message, in primary language, “follow us” mechanisms pre¬ 
sent 

The two alerts appear to be coordinated, or contain the same information. 

The messages are clear and understandable with no spelling or grammar errors 
The messages are written in your primary language 

The EMAs have alternate communications channels (e.g., web site, Facebook page, Twitter ac¬ 
count) that you can access easily. 

How likely are you to understand these alerts? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 
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Case 2: coordinated alerts, clear message, not in primary language, “follow us” mechanisms 
absent 

The two alerts appear to be coordinated, or contain the same information. 

The messages are clear and understandable with no spelling or grammar errors 

The messages are issued in a language that you understand, but one that is not your primary lan¬ 
guage. 

The EMAs have no alternate communications channels (e.g., web site, Facebook page, Twitter ac¬ 
count) that you can access easily. 

How likely are you to understand these alerts? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 


Case 3: coordinated alerts, unclear message, in primary language, “follow us” mechanisms ab¬ 
sent 

The two alerts appear to be coordinated, or contain the same information. 

The messages are not clear and contain some spelling and grammar errors 
The messages are written in your primary language 

The EMAs have no alternate communications channels (e.g., web site, Facebook page, Twitter ac¬ 
count) that you can access easily. 

How likely are you to understand these alerts? (Please choose one) 

□ □ □ □ □ □ □ 


Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

Not 

Not 



Probably 


(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 
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Case 4: coordinated alerts, unclear message, not in primary language, “follow us” mechanisms 
present 

The two alerts appear to be coordinated, or contain the same information. 

The messages are not clear and contain some spelling and grammar errors 

The messages are issued in a language that you understand, but one that is not your primary lan¬ 
guage. 

The EMAs have alternate communications channels (e.g., web site, Facebook page, Twitter ac¬ 
count) that you can access easily. 

How likely are you to understand these alerts? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 


Case 5: uncoordinated alerts, clear message, in primary language, “follow us” mechanisms ab¬ 
sent 

The two alerts are uncoordinated, and do not agree. 

The messages are clear and understandable with no spelling or grammar errors 
The messages are written in your primary language 

The EMAs have no alternate communications channels (e.g., web site, Facebook page, Twitter ac¬ 
count) that you can access easily. 

How likely are you to understand these alerts? (Please choose one) 

□ □ □ □ □ □ □ 


Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

Not 

Not 



Probably 


(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 
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Case 6: uncoordinated alerts, clear message, not in primary language, “follow us” mechanisms 
present 

The two alerts are uncoordinated, and do not agree. 

The messages are clear and understandable with no spelling or grammar errors 

The messages are issued in a language that you understand, but one that is not your primary lan¬ 
guage. 

The EMAs have alternate communications channels (e.g., web site, Facebook page, Twitter ac¬ 
count) that you can access easily. 

How likely are you to understand these alerts? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 


Case 7: uncoordinated alerts, unclear message, in primary language, “follow us” mechanisms 
present 

The two alerts are uncoordinated, and do not agree. 

The messages are not clear and contain some spelling and grammar errors 
The messages are written in your primary language 

The EMAs have alternate communications channels (e.g., web site, Facebook page, Twitter ac¬ 
count) that you can access easily. 

How likely are you to understand these alerts? (Please choose one) 

□ □ □ □ □ □ □ 


Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

Not 

Not 



Probably 


(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 
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Case 8: uncoordinated alerts, unclear message, not in primary language, “follow us” mecha¬ 
nisms absent 

The two alerts are uncoordinated, and do not agree. 

The messages are not clear and contain some spelling and grammar errors 

The messages are issued in a language that you understand, but one that is not your primary lan¬ 
guage. 

The EMAs have no alternate communications channels (e.g., web site, Facebook page, Twitter ac¬ 
count) that you can access easily. 

How likely are you to understand these alerts? (Please choose one) 

□ □ □ □ □ □ □ 


Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

Not 

Not 



Probably 


(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 
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Scenario 7. This set of scenarios probes the impact of an alert message’s specification of what 

has happened and why you should act, the relevance of the message, the message 
clarity, and the ability to confirm via social media on your willingness to believe an 
alert 

Your city’s Emergency Management Agency (EMA) has been using WEA to alert the public to 
imminent threats for the past 12 months. You have received WEA alerts on you mobile phone 
over the past year. 

At 4:37 PM your mobile phone receives and displays an alert. 

For each of the following cases, please indicate how likely you are to believe this alert. 


Case 1: includes “what, ” includes “why, ” relevant, clear message, not confirmed via social me- 
_ dm _ 

“Nuclearplant radiation release 

in this area 

Take shelter now 

to avoid radiation exposure ’’ 

You determine that the alert is relevant to you (e.g., it applies to your location, it is issued in a 
timely manner, it addresses an emergency that affects you). 

You cannot find confirmation of the information on social media (e.g., Twitter, Facebook) 

How likely are you to believe this alert? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 
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Case 2: includes “what, ” includes “why, ” irrelevant, unclear message, confirmed via social me- 
_ dm _ 

“Nucularplant radiatin releese 

in this area 

Take sheltar now 

to avoid radiatin expossur" 

You determine that the alert is not relevant to you (e.g., it does not apply to your location, it is not 
issued in a timely manner, it does not address an emergency that affects you). 

You can find confirmation of the information on social media (e.g., Twitter, Facebook) 

How likely are you to believe this alert? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 


Case 3: includes “what, ” omits “why, ” relevant, unclear message, confirmed via social media 

“Nucularplant radiatin releese 

in this area 

Take sheltar now ” 

You determine that the alert is relevant to you (e.g., it applies to your location, it is issued in a 
timely manner, it addresses an emergency that affects you). 

You can find confirmation of the information on social media (e.g., Twitter, Facebook) 

How likely are you to believe this alert? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 


Case 4: includes “what, ” omits “why, ” irrelevant, clear message, unconfirmed via social media 

“Nuclearplant radiation release 

in this area 

Take shelter now ” 

You determine that the alert is not relevant to you (e.g., it does not apply to your location, it is not 
issued in a timely manner, it does not address an emergency that affects you). 

You cannot find confirmation of the information on social media (e.g., Twitter, Facebook) 

How likely are you to believe this alert? (Please choose one) 
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□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 


Case 5: omits “what, ” includes “why, ” relevant, unclear message, unconfirmed via social media 

“Nucular power plant warnign 

in this area 

Take sheltar now 

to avoid radiatin expossur" 

You determine that the alert is relevant to you (e.g., it applies to your location, it is issued in a 
timely manner, it addresses an emergency that affects you). 

You cannot find confirmation of the information on social media (e.g., Twitter, Facebook) 

How likely are you to believe this alert? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 


Case 6: omits “what, ” includes “why, ” irrelevant, clear message, confirmed via social media 

“Nuclear Power plant warning 

in this area 

Take shelter now 

to avoid radiation exposure ” 

You determine that the alert is not relevant to you (e.g., it does not apply to your location, it is not 
issued in a timely manner, it does not address an emergency that affects you). 

You can find confirmation of the information on social media (e.g., Twitter, Facebook) 

How likely are you to believe this alert? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 


CMU/SEI-2013-SR-021 | 260 




Case 7: omits “what, ” omits “why, ” relevant, clear message, confirmed via social media 

"Nuclear Power plant warning 

in this area 

Take shelter now ’’ 

You determine that the alert is relevant to you (e.g., it applies to your location, it is issued in a 
timely manner, it addresses an emergency that affects you). 

You can find confirmation of the information on social media (e.g., Twitter, Facebook) 

How likely are you to believe this alert? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 


Case 8: omits “what, ” omits “why, ” irrelevant, unclear message, unconfirmed via social media 

“Nucular Power plant warnign 

in this area 

Take sheltar now ” 

You determine that the alert is not relevant to you (e.g., it does not apply to your location, it is not 
issued in a timely manner, it does not address an emergency that affects you). 

You cannot find confirmation of the information on social media (e.g., Twitter, Facebook) 

How likely are you to believe this alert? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 
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Scenario 8. This set of scenarios probes the impact of the ability to confirm the alert from other 

sources, the history of relevance ofprior alerts, coordination of alerts across multiple 
jurisdictions, and public outreach on your willingness to believe an alert. 

Your city’s Emergency Management Agency (EMA) has been using WEA to alert the public to 
imminent threats for the past 12 months. You have received WEA alerts on you mobile phone 
over the past year. 

At 4:37 PM your mobile phone receives and displays an alert from your city EMA regarding an 
imminent threat (i.e., a flash flood) in your area. At 4:48 your mobile phone receives and displays 
an alert from your county EMA regarding the same emergency. 

For each of the following cases, please indicate how likely you are to believe these alerts. 


Case 1: confirmed, history of relevance, coordinated, public outreach 
You can confirm alert information via other channels such as radio or TV news. 

You believe that most of the prior alerts that you have received were relevant to you. 

The two alerts appear to be coordinated, or contain the same information. 

You have previously been made aware of the WEA program via public outreach (e.g., newspaper 
articles, mobile service provider mailings, TV and radio news stories) from the EMAs. 

How likely are you to believe these alerts? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 


Case 2: confirmed, history of relevance, uncoordinated, no public outreach 
You can confirm alert information via other channels such as radio or TV news. 

You believe that most of the prior alerts that you have received were relevant to you. 

The two alerts are uncoordinated, and do not agree. 

You have not been made aware of the WEA program via public outreach (e.g., newspaper articles, 
mobile service provider mailings, TV and radio news stories) from the EMAs. 

How likely are you to believe these alerts? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 
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Case 3: confirmed, history of irrelevance, coordinated, no public outreach 


You can confirm alert information via other channels such as radio or TV news. 

You believe that most of the prior alerts that you have received were not relevant to you. 

The two alerts appear to be coordinated, or contain the same information. 

You have not been made aware of the WEA program via public outreach (e.g., newspaper articles, 
mobile service provider mailings, TV and radio news stories) from the EMAs. 

How likely are you to believe these alerts? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 


Case 4: confirmed, history of irrelevance, uncoordinated, public outreach 
You can confirm alert information via other channels such as radio or TV news. 

You believe that most of the prior alerts that you have received were not relevant to you. 

The two alerts are uncoordinated, and do not agree. 

You have previously been made aware of the WEA program via public outreach (e.g., newspaper 
articles, mobile service provider mailings, TV and radio news stories) from the EMAs. 

How likely are you to believe these alerts? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 
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Case 5: unconfirmed, history of relevance, coordinated, no public outreach 


You can find no confirmation of the alert information via other channels such as radio or TV 
news. 

You believe that most of the prior alerts that you have received were relevant to you. 

The two alerts appear to be coordinated, or contain the same information. 

You have not been made aware of the WEA program via public outreach (e.g., newspaper articles, 
mobile service provider mailings, TV and radio news stories) from the EMAs. 

How likely are you to believe these alerts? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 


Case 6: unconfirmed, history of relevance, uncoordinated, public outreach 

You can find no confirmation of the alert information via other channels such as radio or TV 
news. 

You believe that most of the prior alerts that you have received were relevant to you. 

The two alerts are uncoordinated, and do not agree. 

You have previously been made aware of the WEA program via public outreach (e.g., newspaper 
articles, mobile service provider mailings, TV and radio news stories) from the EMAs. 

How likely are you to believe these alerts? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 
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Case 7: unconfirmed, history of irrelevance, coordinated, public outreach 

You can find no confirmation of the alert information via other channels such as radio or TV 
news. 

You believe that most of the prior alerts that you have received were not relevant to you. 

The two alerts appear to be coordinated, or contain the same information. 

You have previously been made aware of the WEA program via public outreach (e.g., newspaper 
articles, mobile service provider mailings, TV and radio news stories) from the EMAs. 

How likely are you to believe these alerts? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 


Case 8: unconfirmed, history of irrelevance, uncoordinated, no public outreach 

You can find no confirmation of the alert information via other channels such as radio or TV 
news. 

You believe that most of the prior alerts that you have received were not relevant to you. 

The two alerts are uncoordinated, and do not agree. 

You have not been made aware of the WEA program via public outreach (e.g., newspaper articles, 
mobile service provider mailings, TV and radio news stories) from the EMAs. 

How likely are you to believe these alerts? (Please choose one) 

□ □ □ □ □ □ □ 


Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

Not 

Not 



Probably 


(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 
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Scenario 9. This set of scenarios probes the impact of the issuance of a final “all clear” alert, the 
source of the alert, the frequency of alerts, and provision of references for more in¬ 
formation on your willingness to believe an alert. 

Your city’s Emergency Management Agency (EMA) has been using WEA to alert the public to 
imminent threats for the past 12 months. You have received WEA alerts on you mobile phone 
over the past year. 

At 4:37 PM your mobile phone receives and displays an alert regarding an imminent threat (i.e., a 
flash flood) in your area. 

For each of the following cases, please indicate how likely you are to believe this alert. 


Case 1: final communication, local alert source, few prior alerts, references for more info 

Prior WEA alerts that you have received have included an “all clear” message after the event has 
been resolved? 

The alert addresses an imminent threat to your area and is sent by your local Emergency Manage¬ 
ment Agency 

Over the past 6 months, you have received few alerts - 6 from the National Weather Service re¬ 
garding severe weather events, and 1 AMBER alert, and 2 alerts from your city’s EMA regarding 
local threats. 

The message includes a recommendation to “Check TV & Radio for more info” 

How likely are you to believe this alert? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 
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Case 2: final communication, local alert source, many prior alerts, no references for more info 


Prior WEA alerts that you have received have included an “all clear” message after the event has 
been resolved? 

The alert addresses an imminent threat to your area and is sent by your local Emergency Manage¬ 
ment Agency 

Over the past 6 months, you have received many alerts - 48 from the National Weather Service re¬ 
garding severe weather events, and 12 AMBER alerts, and 18 alerts from your city’s EMA regard¬ 
ing local threats. 

The message does not include a recommendation to “Check TV & Radio for more info” 

How likely are you to believe this alert? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 


Case 3: final communication, national alert source, few prior alerts, no references for more in¬ 

fo 

Prior WEA alerts that you have received have included an “all clear” message after the event has 
been resolved? 

The alert addresses an imminent threat to your area and is sent by a national government agency 
(e.g., FBI, DHS, FEMA). 

Over the past 6 months, you have received few alerts - 6 from the National Weather Service re¬ 
garding severe weather events, and 1 AMBER alert, and 2 alerts from your city’s EMA regarding 
local threats. 

The message does not include a recommendation to “Check TV & Radio for more info” 


How likely are you to believe this alert? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

Not 

Not 



Probably 


(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 
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Case 4: final communication, national alert source, many prior alerts, references for more info 


Prior WEA alerts that you have received have included an “all clear” message after the event has 
been resolved? 

The alert addresses an imminent threat to your area and is sent by a national government agency 
(e.g., FBI, DHS, FEMA). 

Over the past 6 months, you have received many alerts - 48 from the National Weather Service re¬ 
garding severe weather events, and 12 AMBER alerts, and 18 alerts from your city’s EMA regard¬ 
ing local threats. 

The message includes a recommendation to “Check TV & Radio for more info” 

How likely are you to believe this alert? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 


Case 5: no final communication, local alert source, few prior alerts, no references for more info 

Prior WEA alerts that you have received have not included an “all clear” message after the event 
has been resolved? 

The alert addresses an imminent threat to your area and is sent by your local Emergency Manage¬ 
ment Agency 

Over the past 6 months, you have received few alerts - 6 from the National Weather Service re¬ 
garding severe weather events, and 1 AMBER alert, and 2 alerts from your city’s EMA regarding 
local threats. 

The message does not include a recommendation to “Check TV & Radio for more info” 

How likely are you to believe this alert? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 
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Case 6: no final communication, local alert source, many prior alerts, references for more info 


Prior WEA alerts that you have received have not included an “all clear” message after the event 
has been resolved? 

The alert addresses an imminent threat to your area and is sent by your local Emergency Manage¬ 
ment Agency 

Over the past 6 months, you have received many alerts - 48 from the National Weather Service re¬ 
garding severe weather events, and 12 AMBER alerts, and 18 alerts from your city’s EMA regard¬ 
ing local threats. 

The message includes a recommendation to “Check TV & Radio for more info” 


How likely are you to believe this alert? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Not 

Very Probably 

Not 

Probably 

Not 

Maybe 

Probably 

Very 

Probably 

Definitely 

(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 


Case 7: no final communication, national alert source, few prior alerts, references for more in¬ 

fo 

Prior WEA alerts that you have received have not included an “all clear” message after the event 
has been resolved? 

The alert addresses an imminent threat to your area and is sent by a national government agency 
(e.g., FBI, DHS, FEMA). 

Over the past 6 months, you have received few alerts - 6 from the National Weather Service re¬ 
garding severe weather events, and 1 AMBER alert, and 2 alerts from your city’s EMA regarding 
local threats. 

The message includes a recommendation to “Check TV & Radio for more info” 


How likely are you to believe this alert? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

Not 

Not 



Probably 


(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 
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Case 8: no final communication, national alert source, many prior alerts, no references for 
more info 

Prior WEA alerts that you have received have not included an “all clear” message after the event 
has been resolved? 

The alert addresses an imminent threat to your area and is sent by a national government agency 
(e.g., FBI, DHS, FEMA). 

Over the past 6 months, you have received many alerts - 48 from the National Weather Service re¬ 
garding severe weather events, and 12 AMBER alerts, and 18 alerts from your city’s EMA regard¬ 
ing local threats. 

The message does not include a recommendation to “Check TV & Radio for more info” 

How likely are you to believe this alert? (Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

Not 

Not 



Probably 


(<5%) 

(5-20%) 

(20-40%) 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 
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Appendix W Public Trust Model Validation Graphical Results 


In each figure, the vertical axis shows the number of respondents who chose a particular answer. 
The horizontal axis refers to the answer selected by the respondents: 1 = Definitely Not, 2 = Very 
Probably Not, 3 = Probably Not, 4 = Maybe, 5 = Probably, 6 = Very Probably, 7 = Definitely. 

Scenario 1 


e 

s 



Figure 45: Case 1: Includes “What" and “Why” and Omits “Who" 



Figure 46: Case 2: Includes “What,”Omits “Why,”and Includes “Who” 



Figure 47: Case 3: Omits “What”and Includes “Why” and “Who” 
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Scenario 2 



Figure 49: Case 1: Defined Action and Time, Sufficient Lead, tnctudes “Who,” and trreievant 



1 2 3 4 S 6 7 


Figure 50: Case 2: Defined Action and Time, Insufficient Lead, Omits “Who,” and Relevant 


4.5 

4 • 

3.5 • 

3 • 

2.5 
2 

1.5 • 

1 • 

0.5 • 

0 • 

12 3 4 



5 6 7 


Figure 51: Case 3: Defined Action, Undefined Time, Sufficient Lead, Omits “Who,” and Relevant 
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Figure 52: Case 4: Defined Action, Undefined Time, insufficient Lead, tnctudes “Who, ” and Irreievant 

Scenario 3 



Figure 53: Case 5: Undefined Action, Defined Time, Sufficient Lead, Omits “Who, ” and Irreievant 



Figure 54: Case 6: Undefined Action, Defined Time, Insufficient Lead, Includes “Who,” and Relevant 



Figure 55: Case 7: Undefined Action and Time, Sufficient Lead, Includes “Who, ” and Relevant 
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Figure 56: Case 8: Undefined Action and Time, Insufficient Lead, Omits “Who, ” and Irrelevant 


Scenario 3 



Figure 57: Case 1: Imminent Threat, Few Prior Alerts, Sufficient Public Outreach, and History of Rele¬ 
vance 



Figure 58: Case 2: Imminent Threat, Few Prior Alerts, Insufficient Public Outreach, and History of Irrele¬ 
vance 
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Figure 59: Case 3: Imminent Threat, Many Prior Alerts, Sufficient Public Outreach, and History of Irrele¬ 
vance 



Figure 60: Case 4: Imminent Threat, Many Prior Alerts, Insufficient Public Outreach, and History of Rel¬ 
evance 



Figure 61: Case 5: AMBER Alert, Few Prior Alerts, Sufficient Public Outreach, and History of Irrele¬ 
vance 
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Figure 62: Case 6: AMBER Alert, Few Prior Alerts, Insufficient Public Outreach, and History of Rele¬ 
vance 



Figure 63: Case 7: AMBER Alert, Many Prior Alerts, Sufficient Public Outreach, and History of Rele¬ 
vance 



Figure 64: Case 8: AMBER Alert, Many Prior Alerts, Insufficient Public Outreach, and History of Irrele¬ 
vance 
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Scenario 4 
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Figure 65: Case 1: Relevant, Clear, and Unconfirmed 



Figure 66: Case 2: Relevant, Unclear, and Confirmed 



Figure 67: Case 3: Irrelevant, Clear, and Confirmed 
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Figure 68: Case 4: Irrelevant, Unclear, and Unconfirmed 


Scenario 5 



Figure 69: Case 1: Coordinated Alerts, Confirmed, and Considered Spam 



Figure 70: Case 2: Coordinated Alerts, Unconfirmed, and Not Considered Spam 
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Figure 71: Case 3: Uncoordinated Alerts, Confirmed, and Not Considered Spam 



Figure 72: Case 4: Uncoordinated Alerts, Unconfirmed, and Considered Spam 

Scenario 6 



Figure 73: Case 1: Coordinated Alerts, Clear Message, in Primary Language, and “Follow-Us” Mecha¬ 
nisms Present 
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Figure 74: Case 2: Coordinated Alerts, Clear Message, Not in Primary Language, and ToHow-Us” 
Mechanisms Absent 



Figure 75: Case 3: Coordinated Alerts, Unclear Message, in Primary Language, and “Follow-Us” Mech¬ 
anisms Absent 
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Figure 76: Case 4: Coordinated Alerts, Unclear Message, Notin Primary Language, and ‘Follow-Us” 
Mechanisms Present 
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Figure 77: Case 5: Uncoordinated Aierts, dear Message, in Primary Language, and “Foiiow-Us” Mech¬ 
anisms Absent 



Figure 78: Case 6: Uncoordinated Aierts, dear Message, in Primary Language, and “Foiiow-Us" Mech¬ 
anisms Absent 



Figure 79: Case 7: Uncoordinated Aierts, dear Message, in Primary Language, and “Foiiow-Us” Mech¬ 
anisms Absent 
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Figure 80: Case 8: Uncoordinated Aierts, Unciear Message, Not in Primary Language, and “Foiiow-Us” 
Mechanisms Absent 

Scenario 7 



Figure 81: Case 1: inciudes “What” and “Why, ” Reievant, dear Message, and Not Confirmed via Sociai 
Media 



Figure 82: Case 2: inciudes “What” and “Why, ” trreievant, Unciear Message, and Confirmed via Sociai 
Media 


CMU/SEI-2013-SR-021 | 282 













Figure 83: Case 3: Includes “What, ” Omits “Why, ” Relevant, Unclear Message, and Confirmed via So¬ 
cial Media 



Figure 84: Case 4: Includes “What, ” Omits “Why, ” Irrelevant, Clear Message, and Not Confirmed via 
Social Media 



Figure 85: Case 5: Omits “What, ” Includes “Why, ” Relevant, Unclear Message, and Not Confirmed via 
Social Media 
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Figure 86: Case 6: Omits “What, ” tnciudes “Why, ” trretevant, dear Message, and Confirmed via Sociai 
Media 



Figure 87: Case 7: Omits “What" and “Why,” Reievant, dear Message, and Confirmed via Sociai Media 



Figure 88: Case 8: Omits “What” and “Why, ” trretevant, Unciear Message, and Not Confirmed via Sociai 
Media 


CMU/SEI-2013-SR-021 | 284 










Scenario 8 
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Figure 90: Case 2: Confirmed, History of Relevance, Uncoordinated, and No Public Outreach 



Figure 91: Case 3: Confirmed, History of Irrelevance, Coordinated, and No Public Outreach 
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Figure 95: Case 7: Unconfirmed, History of irrelevance, Coordinated, and Public Outreach 



Figure 96: Case 8: Unconfirmed, History of Irrelevance, Uncoordinated, and No Public Outreach 
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Appendix X Public Trust Model Validation Statistical 
Analysis 


Table 62: Validation of Public BBN Model Against Results of Public Validation Scenario Interviews 


Validation 

BBN 

Factor 

0.875 

0.73 

Relevance 

0.875 

0.56 

Relevance 

0.975 

0.73 

Relevance 

0.5 

0.56 

Relevance 

0.125 

0.54 

Acting 

0.875 

0.59 

Acting 

0.875 

0.6 

Acting 

0.025 

0.53 

Acting 

0.025 

0.52 

Acting 

0.975 

0.6 

Acting 

0.975 

0.6 

Acting 

0.025 

0.51 

Acting 

0.125 

0.39 

ViewSpam 

0.7 

0.42 

ViewSpam 

0.5 

0.34 

ViewSpam 

0.3 

0.35 

ViewSpam 

0.5 

0.4 

ViewSpam 

0.3 

0.41 

ViewSpam 

0.125 

0.34 

ViewSpam 

0.5 

0.35 

ViewSpam 

0.5 

0.36 

ViewSpam 

0.3 

0.32 

ViewSpam 

0.125 

0.45 

ViewSpam 

0.975 

0.57 

ViewSpam 

0.3 

0.5 

OptOut 

0.3 

0.6 

OptOut 

0.125 

0.5 

OptOut 

0.7 

0.5 

OptOut 

0.975 

0.58 

Understand 

0.875 

0.53 

Understand 


Validation 

BBN 

Factor 

0.7 

0.55 

Understand 

0.5 

0.53 

Understand 

0.5 

0.61 

Understand 

0.5 

0.59 

Understand 

0.5 

0.61 

Understand 

0.125 

0.57 

Understand 

0.875 

0.56 

Believe 

0.5 

0.54 

Believe 

0.7 

0.54 

Believe 

0.5 

0.56 

Believe 

0.3 

0.54 

Believe 

0.875 

0.56 

Believe 

0.875 

0.56 

Believe 

0.125 

0.54 

Believe 

0.975 

0.57 

Believe 

0.7 

0.55 

Believe 

0.7 

0.54 

Believe 

0.5 

0.55 

Believe 

0.7 

0.56 

Believe 

0.7 

0.57 

Believe 

0.5 

0.55 

Believe 

0.125 

0.53 

Believe 

0.875 

0.55 

Believe 

0.7 

0.56 

Believe 

0.7 

0.55 

Believe 

0.875 

0.56 

Believe 

0.7 

0.54 

Believe 

0.875 

0.55 

Believe 

0.875 

0.54 

Believe 

0.5 

0.55 

Believe 
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Scatterplot of Val idation vs BBN 



Figure 97: Analysis of All Predictions Across All Outcome Nodes in Table 62 

Regression Analysis: Validation vs. BBN 

The regression equation is Validation = -0.295 + 1.63 BBN. 

Table 63: Regression Coefficient Results for Validation vs. BBN 


Predictor 

Coef 

SE Coef 

t 

P 

Constant 

-0.2955 

0.2195 

-1.35 

0.184 

BBN 

1.6315 

0.4109 

3.97 

0.000 


S = 0.267195, R-Sq = 21.4%, R-Sq(adj) = 20.0% 


Table 64: Analysis of Variance for Validation vs. BBN 


Source 

df 

SS 

MS 

F 

P 

Regression 

1 

1.1256 

1.1256 

15.77 

0.000 

Residual error 

58 

4.1408 

0.0714 



Total 

59 

5.2664 
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Figure 98: Residual Plots for Regression of Overall Validation as Function of BBN Predictions 


Scatterplot of Val idation vs BBN 


0.3 0.5 0.7 



Figure 99: Scatterplots of Six Factors Comparing BBN Predictions to Validation Values 

The above attempt to discover significance with all output node types proved unsuccessful. 
Hence, we performed an analysis by selected factors. 

Act: Regression Analysis 

The regression equation is Acting-V = -6.04 + 11.6 Acting-B. 

Table 65: Regression Coefficient Results for Acting vs. BBN 


Predictor 

Coef 

SE Coef 

t 

P 

Constant 

-6.0362 

0.4914 

-12.28 

0.000 

BBN 

11.6234 

0.8737 

13.30 

0.000 


S = 0.0919945, R-Sq = 96.7%, R-Sq(adj) = 96.2% 
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Table 66: Analysis of Variance forActing-V vs. Acting-B 


Source 

df 

SS 

MS 

F 

P 

Regression 

1 

1.4980 

1.4980 

177.00 

0.000 

Residual error 

6 

0.0508 

0.0085 



Total 

7 

1.5487 





Residual Plots for Acting-V 

Normal Probability Plot 
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Figure 100: 


Residual Plots for Regression of Acting Validation as a Function of the BBN 


Believe: Regression Analysis 

The regression equation is Believe-V = -6.35 + 12.7 Believe-B. 
Table 67: Regression Coefficient Results for Believe vs. BBN 


Predictor 

Coef 

SE Coef 

t 

P 

Constant 

-6.354 

2.127 

-2.99 

0.007 

BBN 

12.726 

3.860 

3.30 

0.003 


S = 0.196209, R-Sq = 33.1%, R-Sq(adj) = 30.0% 


Table 68: Analysis of Variance for Believe-V vs. Believe-B 


Source 

df 

SS 

MS 

F 

P 

Regression 

1 

0.41836 

0.41836 

10.87 

0.003 

Residual error 

22 

0.84695 

0.03850 



Total 

23 

1.26531 
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Residual Plots for Believe-V 

Normal Probability Plot 



Histogram 
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Figure 101: 


Residual Plots for Regression of Believing Validation as a Function of the BBN 


Understand: Regression Analysis 

The regression equation is Understand-V = 1.77 — 2.07 Understand-B. 


Table 69: Regression Coefficient Results for Understand vs. BBN 


Predictor 

Coef 

SE Coef 

t 

P 

Constant 

1.768 

1.853 

0.95 

0.377 

BBN 

-2.071 

3.239 

-0.64 

0.546 


S = 0.276484, R-Sq = 6.4%, R-Sq(adj) = 0.0% 


Table 70: Analysis of Variance for Understand-V vs. Understand-B 


Source 

df 

SS 

MS 

F 

P 

Regression 

1 

0.03126 

0.03126 

0.41 

0.546 

Residual error 

6 

0.45866 

0.07644 



Total 

7 

0.48992 
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Figure 102: Residual Plots for Regression of Understand Validation as a Function of the BBN 

View as Spam: Regression Anaiysis 

The regression equation is ViewSpam-V = -0.407 + 2.09 ViewSpam-B. 

Table 71: Regression Coefficient Results for ViewSpam vs. BBN 


Predictor 

Coef 

SE Coef 

t 

P 

Constant 

-0.4072 

0.3881 

-1.05 

0.319 

BBN 

2.0928 

0.9773 

2.14 

0.058 


S = 0.221506, R-Sq = 31.4%, R-Sq(adj) = 24.6% 


Table 72: Analysis of Variance for ViewSpam-V vs. ViewSpam-B 


Source 

df 

ss 

MS 

F 

P 

Regression 

1 

0.22498 

0.22498 

4.59 

0.058 

Residual error 

10 

0.49065 

0.04906 



Total 

11 

0.71562 
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Residual Plots for ViewSpam-V 




Figure 103: Residual Plots for Regression of ViewSpam Validation as a Function of the BBN 


Opt Out: Regression Analysis 

The regression equation is OptOut-V = 0.75 - 0.75 OptOut-B. 


Table 73: Regression Coefficient Results for OptOut vs. BBN 


Predictor 

Coef 

SE Coef 

t 

P 

Constant 

0.750 

1.793 

0.42 

0.716 

BBN 

-0.750 

3.403 

-0.22 

0.846 


S = 0.294746, R-Sq = 2.4%, R-Sq(adj) = 0.0% 


Table 74: Analysis of Variance for OptOut-V vs. OptOut-B 


Source 

df 

SS 

MS 

F 

P 

Regression 

1 

0.00422 

0.00422 

0.05 

0.846 

Residual error 

2 

0.17375 

0.08688 



Total 

3 

0.17797 
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Residual Plots for OptOut-V 


Normal Probability Plot 



Histogram 





Residual 


Versus Fits 

0.4i- 


'S 0.0 -e- 

oe 

- 0 . 2 - 


0.30 0.32 0.34 0.36 0.38 

Fitted Value 


Versus Order 



Figure 104: Residual Plots for Regression of OptOut Validation as a Function of the BBN 


Relevance: Regression Analysis 

The regression equation is Relevance-V = -0.095 + 1.40 Relevance-B. 


Table 75: Regression Coefficient Results for Relevance vs. BBN 


Predictor 

Coef 

SE Coef 

t 

P 

Constant 

-0.0949 

0.7426 

-0.13 

0.910 

BBN 

1.397 

1.141 

1.22 

0.346 


S = 0.194052, R-Sq = 42.8%, R-Sq(adj) = 14.2% 


Table 76: Analysis of Variance for Relevance-V vs. Relevance-B 


Source 

df 

SS 

MS 

F 

P 

Regression 

1 

0.05641 

0.05641 

1.50 

0.346 

Residual error 

2 

0.07531 

0.03766 



Total 

3 

0.13172 
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Residual Plots for Relevance-V 


Normal Probability Plot 
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Figure 105: 


Residual Plots for Regression of Relevance Validation as a Function of the BBN 


Understand Plus Believe Plus Act: Regression Analysis 

The regression equation is Validation = -2.52 + 5.61 BBN. 


Table 77: Regression Coefficient Results for Understand, Believe, and Act vs. BBN 


Predictor 

Coef 

SE Coef 

t 

P 

Constant 

-2.5191 

0.9729 

-2.59 

0.014 

BBN 

5.614 

1.745 

3.2 

0.003 


S = 0.268294, R-Sq = 21.4%, R-Sq(adj) = 19.3% 


Table 78: Analysis of Variance for Understand, Believe, and Act vs. BBN 


Source 

df 

SS 

MS 

F 

P 

Regression 

1 

0.74517 

0.74517 

10.35 

0.003 

Residual error 

38 

2.73531 

0.07198 



Total 

39 

3.48048 
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Scatterplot of Validation vs BBN 



Figure 106: Scatterplot of BBN Predictions vs. Validation for Understand, Believe, and Act Only 


Residual Plots for Validation 
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Figure 107 ; Residual Plots for Regression of Understand, Believe, and Act Only Validation as a 
Function of the BBN 


CMU/SEI-2013-SR-021 | 297 






















































Appendix Y Alert Originator Trust Model Validation Scenarios and Responses 


Response Key 

1 = Definitely not (<5%) 

2 = Very probably not (5-20%) 

3 = Probably not (20^0%) 

4 = Maybe (40-60%) 

5 = Probably (60-80%) 

6 = Very probably (80-95%) 

7 = Definitely (>95%) 
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Table 79: Alert Originator Trust Model Validation Data 


Scenario 

Factors 

Responses 

Context 

Case 

Description 

Training 

Cybersecurity 

Governance 


< 


CM 

CQ 

u 

CM 

O 

Q 

u] 


CM 

LJ. 

Your agency chose to deploy 
WEA to issue public alerts 16 
months ago. The system is 
fully functional, and through 
training and practice you are 
fully capable of issuing alerts. 
Thunderstorms have been 
occurring across your city for 
the past 24 hours. NWS has 
responsibility for generating 
alerts for weather-related 
events and has issued an alert 
for flash flooding of several 
low-lying areas within the city. 
You receive reliable and con¬ 
firmed information that a dam 
break on the Little Bear reser¬ 
voir is imminent. This will result 
in major flooding downstream. 
For each case below, please 
indicate how likely you are to 
use WEA to issue a public alert 
under the stated circumstanc¬ 
es? 

Case 1: sufficient 
training, sufficient 
cybersecurity, 
insufficient gov¬ 
ernance 

You have been fuliv trained on the use of WEA, and vou 
understand both the principles behind WEA and the op¬ 
eration of the alerting software used in your agency. You 
have successfully practiced issuing alerts with your alert¬ 
ing software. You have templates for alerts that can be 
applied to this event. 

1 

1 

0 


6 

6 

7 

7 

7 

6 

6 

7 

7 

Over the oast 12 months, no cvbersecuritv breaches of 
the WEA service (e.g., access by unauthorized person¬ 
nel, issuance of false alerts) have occurred in your agen¬ 
cy, and none have been reported by other agencies. 

You have an agreement with the local NWS office regard¬ 
ing the division of responsibilities for alerting. This 
agreement clearly defines that NWS will issue alerts for 
all weather-related emergencies, and your agency will 
issue alerts for all other imminent threats. However, since 
the imminent dam break is the result of the ongoing thun¬ 
derstorms, vou are unsure of vour resoonsibilitvfor issu- 
ing an alert for this threat. 
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Scenario 

Factors 

Responses 

Context 

Case 

Description 

Training 

Cybersecurity 

Governance 


< 


CM 

CQ 

u 

CM 

O 

Q 

u] 


CM 

U. 

(see previous page) 

Case 2: sufficient 
training, insuffi¬ 
cient cybersecuri¬ 
ty, sufficient 
governance 

You have been fuliv trained on the use of WEA, and vou 
understand both the principles behind WEA and the op¬ 
eration of the alerting software used in your agency. You 
have successfully practiced issuing alerts with your alert¬ 
ing software. You have templates for alerts that can be 
applied to this event. 

Over the oast 12 months, several cvbersecuritv breaches 
of the WEA service (e.g., access by unauthorized per¬ 
sonnel, issuance of false alerts) have occurred in your 
agency, and others have been reported by other agen¬ 
cies. 

You have an agreement with the local NWS office regard¬ 
ing the division of responsibilities for alerting. This 
agreement clearly defines that NWS will issue alerts for 
all weather -related emergencies, and your agency will 
issue alerts for all other imminent threats. Since the im¬ 
minent dam break is not a weather event, vou are sure of 
vour resDonsibilitv for issuing an alert for this threat. 

1 

0 

1 


6 

7 

7 

7 

7 

6 

3 

7 

7 
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Scenario 

Factors 

Responses 

Context 

Case 

Description 

Training 

Cybersecurity 

Governance 


< 


CM 

CQ 

u 

CM 

O 

Q 

u] 


CM 

U. 

(see previous page) 

Case 3: insufficient 
training, sufficient 
cybersecurity, 
sufficient govern¬ 
ance 

You have not been fuliv trained on the use of WEA, and 
you have only a basic understanding of both the princi¬ 
ples behind WEA and the operation of the alerting soft¬ 
ware used in your agency. You have not had 
opportunities to practice issuing alerts with your alerting 
software. You do not have templates for alerts that can be 
applied to this event. 

Over the oast 12 months, no cvbersecuritv breaches of 
the WEA service (e.g., access by unauthorized person¬ 
nel, issuance of false alerts) have occurred in your agen¬ 
cy, and none have been reported by other agencies. 

You have an agreement with the local NWS office regard¬ 
ing the division of responsibilities for alerting. This 
agreement clearly defines that NWS will issue alerts for 
all weather-related emergencies, and your agency will 
issue alerts for all other imminent threats. Since the im¬ 
minent dam break is not a weather event, vou are sure of 
vour resDonsibilitv for issuing an alert for this threat. 

0 

1 

1 


3 

6 

7 

6 

6 

5 

5 

3 

4 
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Scenario 

Factors 

Responses 

Context 

Case 

Description 

Training 

Cybersecurity 

Governance 


< 


CM 

CQ 

u 

CM 

O 

Q 

u] 


CM 

U. 

(see previous page) 

Case 4: insufficient 
training, insuffi¬ 
cient cybersecuri¬ 
ty, insufficient 
governance 

You have not been fuliv trained on the use of WEA, and 
you have only a basic understanding of both the princi¬ 
ples behind WEA and the operation of the alerting soft¬ 
ware used in your agency. You have not had 
opportunities to practice issuing alerts with your alerting 
software. You do not have templates for alerts that can be 
applied to this event. 

Over the oast 12 months, several cvbersecuritv breaches 
of the WEA service (e.g., access by unauthorized per¬ 
sonnel, issuance of false alerts) have occurred in your 
agency, and others have been reported by other agen¬ 
cies. 

You have an agreement with the local NWS office regard¬ 
ing the division of responsibilities for alerting. This 
agreement clearly defines that NWS will issue alerts for 
all weather-related emergencies, and your agency will 
issue alerts for all other imminent threats. However, since 
the imminent dam break is the result of the ongoing thun¬ 
derstorms, vou are unsure of vour resoonsibilitvfor issu- 
ing an alert for this threat. 

0 

0 

0 


1 

4 

6 

6 

6 

2 

3 

1 

3 
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Scenario 

Factors 

Responses 

Context 

Case 

Description 

Public Feedback 

Public Outreach 

Alert Frequency 


< 


CM 

CQ 

u 

CM 

O 

Q 

u] 


CM 

U. 

Your agency chose to deploy 
WEA to issue public alerts 6 
months ago. The system is 
fully functional, and through 
training and practice you are 
fully capable of issuing alerts. 

For each case below, please 
indicate how likely you are to 
use WEA to issue a public alert 
for an imminent threat under 
the stated circumstances? 

Case 1: favorable 
feedback, public 
outreach, many 
prior alerts 

The public has responded to prior alerts with calls to 9-1- 
1 operators, and calls and emails to the EMA, the mayor, 
and the city council. 47% of the callers expressed ques¬ 
tions about the WEA service (e.g.. What is this service? 
What should 1 do? Why am 1 getting these alerts? How 
does WEA know where 1 am?t. 42% exoressed aobrecia- 
tion for the alerts received, and 11% obiected to receivino 
the alerts. 

Prior to deolovina WEA, vour aaencv performed oublic 
outreach via TV, radio, and newsoaoers to inform the 
public about the upcoming alerting service. 

Within your jurisdiction over the past 6 months, the NWS 
has used WEA to issue 48 weather-related (e.a., torna- 
does, thunderstorms) alerts. No AMBER alerts have been 
issued. Your aaencv has issued 18 alerts for various 
imminent threats (e.a., oolice actions, chemical soills, 
structure fires). 

1 

1 

0 


5 

7 

7 

7 

7 

5 

6 

7 

7 
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Scenario 

Factors 

Responses 

Context 

Case 

Description 

Public Feedback 

Public Outreach 

Alert Frequency 


< 


CM 

CQ 

u 

CM 

O 

Q 

u] 


CM 

U. 

(see previous page) 

Case 2: favorable 
feedback, no pub¬ 
lic outreach, few 
prior alerts 

The public has responded to prior alerts with calls to 9-1- 
1 operators, and calls and emails to the EMA, the mayor, 
and the city council. 47% of the callers expressed ques¬ 
tions about the WEA service (e.g.. What is this service? 
What should 1 do? Why am 1 getting these alerts? How 
does WEA know where 1 am?), 42% exoressed aobrecia- 
tion for the alerts received, and 11% obiected to receivino 
the alerts. 

Your aoencv has not performed oublic outreach via TV, 
radio, and newspapers to inform the public about the 

WEA service. 

Within your jurisdiction over the past 6 months, the NWS 
has used WEA to issue 8 weather-related (e.a., torna- 
does, snowstorms) alerts. No AMBER alerts have been 
issued. Your aoencv has issued 3 alerts for imminent 
threats: one for a oolice action (armed standoff), one for a 
chemical spill, and one for a fire requiring a 4-square- 
block evacuation. 

1 

0 

1 


6 

7 

7 

7 

7 

5 

6 

7 

7 
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Scenario 

Factors 

Responses 

Context 

Case 

Description 

Public Feedback 

Public Outreach 

Alert Frequency 


< 


CM 

CQ 

u 

CM 

O 

Q 

u] 


CM 

U. 

(see previous page) 

Case 3: unfavora- 
ble feedback, pub¬ 
lic outreach, few 
prior alerts 

The public has responded to prior alerts with calls to 9-1- 
1 operators, and calls and emails to the EMA, the mayor, 
and the city council. 47% of the callers expressed ques¬ 
tions about the WEA service (e.g., What is this service? 
What should 1 do? Why am 1 getting these alerts? How 
does WEA know where 1 am?), 11% expressed apprecia¬ 
tion for the alerts received, and 42% obiected to receivino 
the alerts. 

Prior to deolovina WEA, vour aaencv performed oublic 
outreach via TV, radio, and newsoaoers to inform the 
public about the WEA service. 

Within your jurisdiction over the past 6 months, the NWS 
has used WEA to issue 8 weather-related (e.a., torna- 
does, snowstorms! alerts. No AMBER alerts have been 
issued. Your aaencv has issued 3 alerts for imminent 
threats: one for a oolice action (armed standoff!, one for a 
chemical spill, and one for a fire requiring a 4-square- 
block evacuation. 

0 

1 

1 


5. 

5 

7 

7 

5 

5 

2 

5 

7 

7 
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Scenario 

Factors 

Responses 

Context 

Case 

Description 

Public Feedback 

Public Outreach 

Alert Frequency 


< 


CM 

CQ 

u 

CM 

O 

Q 

u] 


CM 

U. 

(see previous page) 

Case 4: unfavora- 
ble feedback, no 
public outreach, 
many prior alerts 

The public has responded to prior alerts with calls to 9-1- 
1 operators, and calls and emails to the EMA, the mayor, 
and the city council. 47% of the callers expressed ques¬ 
tions about the WEA service (e.g., What is this service? 
What should 1 do? Why am 1 getting these alerts? How 
does WEA know where 1 am?), 11% expressed apprecia¬ 
tion for the alerts received, and 42% obiected to receivina 
the alerts. 

0 

0 

0 


5 

5 

7 

5 

5 

2 

4 

7 

7 

Your aoencv has not performed oublic outreach via TV, 
radio, and newspapers to inform the public about the 

WEA service. 

Within your jurisdiction over the past 6 months, the NWS 
has used WEA to issue 48 weather-related (e.a., torna- 
does, thunderstorms) alerts. No AMBER alerts have been 
issued. Your aoencv has issued 18 alerts for various 
imminent threats (e.o., police actions, chemical spills, 
structure fires). 
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Scenario 

Factors 

Responses 

Context 

Case 

Description 

Availability 

Accessibility 

Reliability 

Ease of use 

< 


CM 

CQ 

o 

CM 

O 

Q 

u] 


CM 

U. 

Your agency chose to deploy 
WEA to issue public alerts 6 
months ago. The system is 
fully functional, and through 
training and practice you are 
fully capable of issuing alerts. 

For each case below, please 
indicate how likely you are to 
use WEA to issue a public alert 
for an imminent threat under 
the stated circumstances. 

Case 1: high avail¬ 
ability, high acces- 
sibility, high 
reliability, easy to 
use 

Over the past 12 months, the WEA service has been 
down for maintenance approximately 1 hour oer vear. 

WEA alerts can be aenerated and issued from vour office 
or from vour smartphone usina a secure aob that inter- 
faces to your alert generation software. 

Your alertina software and the WEA service successfullv 
transmits 99.9% of alerts that are correctiv confioured and 
entered. 

The process of using your alerting software to create and 
send an alert to IPAWS-OPEN requires approximately 5 
minutes of effort. 

1 

1 

1 

1 

7 

7 

7 

7 

7 

6 

7 

7 

7 

Case 2: high avail¬ 
ability, high acces- 
sibility, low 
reliability, difficult 
to use 

Over the past 12 months, the WEA service has been 
down for maintenance approximately 1 hour oer vear. 

WEA alerts can be aenerated and issued from vour office 
or from vour smartphone usina a secure app that inter- 
faces to your alert generation software. 

Your alertina software and the WEA service successfullv 
transmit 90% of alerts that are correctiv confioured and 
entered. 

The process of using your alerting software to create and 
send an alert to IPAWS-OPEN requires approximately ^ 
minutes of effort. 

1 

1 

0 

0 

4 

5 

4 

4 

4 

2 

2 

3 

4 
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Scenario 

Factors 

Responses 

Context 

Case 

Description 

Availability 

Accessibility 

Reliability 

Ease of use 

< 


CM 

CQ 

o 

CM 

O 

Q 

u] 


CM 

U. 

(see previous page) 

Case 3: high avail- 
ability, low acces¬ 
sibility, high 
reliability, difficult 
to use 

Over the past 12 months, the WEA service has been 
down for maintenance approximately 1 hour oer vear. 

WEA alerts can be aenerated and issued onlv from a 
dedicated terminal within vour aaencv’s central office. 

Your alertina software and the WEA service successfullv 
transmit 99.9% of alerts that are correctiv confiaured and 
entered. 

The process of using your alerting software to create and 
send an alert to IPAWS-OPEN requires approximately 40 
minutes of effort. 

1 

0 

1 

0 

4 

5 

4 

4 

1 

2 

2 

4 

3 

Case 4: high avail- 
ability, low acces¬ 
sibility, low 
reliability, easy to 
use 

Over the past 12 months, the WEA service has been 
down for maintenance approximately 1 hour oer vear. 

WEA alerts can be aenerated and issued onlv from a 
dedicated terminal within vour aaencv’s central office. 

Your alertina software and the WEA service successfullv 
transmit 90% of alerts that are correctiv confiaured and 
entered. 

The process of using your alerting software to create and 
send an alert to IPAWS-OPEN requires approximately 5 
minutes of effort. 

1 

0 

0 

1 

5 

7 

6 

6 

3 

6 

5 

7 

7 
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Scenario 

Factors 

Responses 

Context 

Case 

Description 

Availability 

Accessibility 

Reliability 

Ease of use 

< 


CM 

CQ 

o 

CM 

O 

Q 

u] 


CM 

U. 

(see previous page) 

Case 5: low avail¬ 
ability, high acces- 
sibility, high 
reliability, difficult 
to use 

Over the past 12 months, the WEA service has been 
down for maintenance approximately 1 hour oer week. 

WEA alerts can be aenerated and issued from vour office 
or from vour smartohone usina a secure aoo that inter- 
faces to your alert generation software. 

Your alertina software and the WEA service successfullv 
transmit 99.9% of alerts that are correctiv confiaured and 
entered. 

The process of using your alerting software to create and 
send an alert to IPAWS-OPEN requires approximately ^ 
minutes of effort. 

0 

1 

1 

0 

4 

4 

4 

4 

4 

2 

2 

3 

3 

Case 6: low avail¬ 
ability, high acces- 
sibility, low 
reliability, easy to 
use 

Over the past 12 months, the WEA service has been 
down for maintenance approximately 1 hour oer week. 

WEA alerts can be aenerated and issued from vour office 
or from vour smartohone usina a secure aoo that inter- 
faces to your alert generation software. 

Your alertina software and the WEA service successfullv 
transmit 90% of alerts that are correctiv confiaured and 
entered. 

The process of using your alerting software to create and 
send an alert to IPAWS-OPEN requires approximately 5 
minutes of effort. 

0 

1 

0 

1 

5 

6 

6 

7 

7 

6 

5 

7 

7 
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Scenario 

Factors 

Responses 

Context 

Case 

Description 

Availability 

Accessibility 

Reliability 

Ease of use 

< 


CM 

CQ 

o 

CM 

O 

Q 

u] 


CM 

U. 

(see previous page) 

Case 7: low avail- 
ability, low acces¬ 
sibility, high 
reliability, easy to 
use 

Over the past 12 months, the WEA service has been 
down for maintenance approximately 1 hour oer week. 

WEA alerts can be aenerated and issued onlv from a 
dedicated terminal within vour aaencv’s central office. 

Your alertina software and the WEA service successfullv 
transmit 99.9% of alerts that are correctiv confiaured and 
entered. 

The process of using your alerting software to create and 
send an alert to IPAWS-OPEN requires approximately 5 
minutes of effort. 

0 

0 

1 

1 

5 

6 

7 

7 

4 

6 

5 

7 

7 

Case 8: low avail- 
ability, low acces¬ 
sibility, low 
reliability, difficult 
to use 

Over the past 12 months, the WEA service has been 
down for maintenance approximately 1 hour oer week. 

WEA alerts can be generated and issued only from a 
dedicated terminal within your agency’s central office. 

Your alertina software and the WEA service successfullv 
transmit 90% of alerts that are correctiv confiaured and 
entered. 

The process of using your alerting software to create and 
send an alert to IPAWS-OPEN requires approximately ^ 
minutes of effort. 

0 

0 

0 

0 

3 

4 

4 

4 

1 

2 

2 

3 

3 
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Scenario 

Factors 

Responses 

Context 

Case 

Description 

Timeliness 

Understandability 

Accuracy 


< 


CM 

CQ 

u 

CM 

O 

Q 

u] 


CM 

U. 

Your agency chose to deploy 
WEA to issue public alerts 6 
months ago. The system is 
fully functional, and through 
training and practice you are 
fully capable of issuing alerts. 

For each case below, please 
indicate how likely you are to 
use WEA to issue a public alert 
for an imminent threat under 
the stated circumstances. 

Case 1: rapid dis¬ 
semination, high 
understandability, 
low accuracy 

Prior messages have been distributed within 2 minutes of 
your submitting the WEA data input. 

You can craft any message of your choosing with a max- 
imum size of 270 characters. 

Prior WEA messages sent to iPAWS-OPEN have been 
disseminated with sianificant errors in the message data 
(errors that affect the understandability or content of the 
message). 

1 

1 

0 


3 

6 

7 

4 

4 

2 

4 

4 

7 

Case 2: rapid dis¬ 
semination, low 
understandability, 
high accuracy 

Prior messages have been distributed within 2 minutes of 
your submitting the WEA data input. 

You are able to issue onlv “standardized” messages de- 
veloped by the WEA service based on your CAP (Com¬ 
mon Alerting Protocol) inputs. In other words, you may 
not tailor the message. 

Prior WEA messages sent to IPAWS-OPEN have been 
disseminated with no errors in the message data. 

1 

0 

1 


4 

6 

7 

7 

7 

5 

6 

7 

7 

Case 3: slow dis- 
semination, high 
understandability, 
high accuracy 

Prior messages have been distributed within 10 to 30 
minutes of your submitting the WEA data input. 

You can craft any message of your choosing with a max- 
imum size of 270 characters. 

Prior WEA messages sent to IPAWS-OPEN have been 
disseminated with no errors in the message data. 

0 

1 

1 


4 

5 

4 

5 

5 

5 

4 

7 

7 
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Scenario 

Factors 

Responses 

Context 

Case 

Description 

Timeliness 

Understandability 

Accuracy 


< 


CM 

CQ 

u 

CM 

O 

Q 

u] 


CM 

U. 

(see previous page) 

Case 4: slow dis¬ 
semination, low 
understandability, 
low accuracy 

Prior messages have been distributed within 10 to 30 
minutes of your submitting the WEA data input. 

You are able to issue onlv “standardized” messaaes de- 
veloped by the WEA service based on your CAP (Com¬ 
mon Alerting Protocol) inputs. In other words, you may 
not tailor the message. 

Prior WEA messages sent to IPAWS-OPEN have been 
disseminated with sianificant errors in the message data 
(errors that affect the understandability or content of the 
message). 

0 

0 

0 


1 

4 

6 

3 

3 

1 

2 

4 

4 
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Scenario 

Factors 

Responses 

Context 

Case 

Description 

Urgency 

Severity 

Certainty 


< 


CM 

CQ 

o 

CM 

O 

Q 

iH 


CM 

U. 

Your agency chose to deploy 
WEA to issue public alerts 6 
months ago. The system is 
fully functional, and through 
training and practice you are 
fully capable of issuing alerts. 

For each case below, please 
indicate how likely you are to 
use WEA to issue a public alert 
for an imminent threat under 
the stated circumstances. 

Case 1: high ur- 
gency, high severi- 
ty, low certainty 

For an event that reauires public action within 10 minutes. 

1 

1 

0 


5 

6 

7 

6 

6 

6 

6 

7 

7 

For an event that ooses extraordinary threat to life. 

For a severe and uraent event that is 30% likely to occur. 

Case 2: high ur- 
gency, low severi- 
ty, high certainty 

For an eyent that reauires public action within 10 minutes. 

1 

0 

1 


3 

6 

7 

5 

5 

5 

5 

1 

3 

For an eyent that ooses possible threat to orooerty. 

For a seyere and uraent eyent that is 90% likely to occur. 

Case 3: low ur- 
gency, high severi- 
ty, high certainty 

For an eyent that reauires public action within 2 hours. 

0 

1 

1 


7 

6 

7 

7 

7 

7 

5 

7 

7 

For an eyent that ooses extraordinary threat to life. 

For a seyere and uraent eyent that is 90% likely to occur. 

Case 4: low ur- 
gency, low severi- 
ty, low certainty 

For an eyent that reauires public action within 2 hours. 

0 

0 

0 


1 

4 

2 

3 

3 

1 

2 

1 

2 

For an eyent that poses possible threat to property. 

For a seyere and urgent eyent that is 30% likely to occur. 
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Scenario 

Factors 

Responses 

Context 

Case 

Description 

Geographic Breadth 




< 


CM 

CQ 

u 

CM 

O 

Q 

u] 


CM 

U. 

Your agency chose to deploy 
WEA to issue public alerts 6 
months ago. The system is 
fully functional, and through 
training and practice you are 
fully capable of issuing alerts. 

Do you think WEA would be an 
appropriate tool to issue a 
public alert for a severe and 
urgent event... 

small event / large 
alert 

... where 30% of the alert recipients are in the hazard 
zone and 70% are outside the zone? 

0 




1 

6 

7 

3 

3 

2 

4 

7 

7 

large event / large 
alert 

... where 90% of the alert recipients are in the hazard 
zone and 10% are outside the zone? 

1 




7 

7 

7 

7 

7 

6 

7 

7 

7 
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Scenario 

Factors 

Responses 

Context 

Case 

Description 

Time of Day 




< 


CM 

CQ 

u 

CM 

O 

Q 

iH 


CM 

U. 

Your agency chose to deploy 
WEA to issue public alerts 6 
months ago. The system is 
fully functional, and through 
training and practice you are 
fully capable of issuing alerts. 

Do you think WEA would be an 
appropriate tool to issue a 
public alert for a severe and 
urgent event... 

@ 10:30 AM 

... that occurs at 10:30 AM ? 

0 




7 

7 

7 

7 

7 

6 

6 

7 

7 

@2:30 AM 

... that occurs at 2:30 AM ? 

1 




7 

7 

7 

7 

7 

6 

6 

6 

7 
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Appendix Z Alert Originator Trust Model Validation Survey 


Wireless Emergency Alerting Trust Model 

Thank you for agreeing to participate in our research regarding the Wireless Emergency Alerting 
(WEA) service. Your assistance is greatly appreciated. 

Background 

The SEI has been tasked by DHS Science and Technology Division to facilitate the adoption of the 
FEMA WEA service by emergency management agencies (EMAs). 

For EMAs to adopt and use WEA, they must be confident that WEA will perform as planned - that it 
will accept the alerts that they create and disseminate them to the public in a timely and accurate fash¬ 
ion. EMAs must trust WEA to enhance public safety. Absent this trust, EMAs are unlikely to make the 
effort to adopt and use WEA. 

A part of the SETs task for DHS is to study this issue of trust. In support of this task, we are building a 
“WEA Trust Model” that will model the interactions between the alert originators, the WEA service, 
and the public to predict alert originators’ willingness to use WEA to issue public alerts. Once com¬ 
pleted, this model will enable us to identify actions that alert originators can apply to maximize both 
their trust in WEA, as well as the public’s trust in WEA It will also identify actions to avoid a decrease 
in trust. In this manner, it will provide guidance for alert originators to aid them in maximizing the 
effectiveness of WEA. 

This model considers many factors influencing trust; such as: 

• WEA performance on prior alerts - were the alerts disseminated accurately to the proper geo¬ 
graphic areas in a timely manner 

• Public response to prior alerts - has feedback from the public indicated that they value the 
WEA alerts 

• Ease of use - can alert originators create and send alerts with a reasonable amount of effort 

The SEI is surveying a large number of EMAs to identify the impact of these factors. Your agency has 
probably already received a request to participate in this survey, and we would value your responses. 
Based on the results of the survey, the SEI will build and calibrate the trust model. However, that is not 
the end of the task. 

To be of value, a model must provide a reasonable representation of reality. In order to validate the 
usefulness and accuracy of our model, we must compare its results with actual scenarios from emer¬ 
gency managers. To accomplish this validation, we request that you provide your responses to the fol¬ 
lowing scenarios. We will submit these same scenarios to the trust model. By comparing the model’s 
responses to yours, we can determine how accurately our model is performing 

This questionnaire 

This questionnaire contains seven simplistic scenarios regarding the use of WEA for public alerting. 
Each scenario addresses several factors that we believe influence trust. Each scenario explores various 
combinations of trust factors as noted below: 
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Scenario 


Trust factors 


1: 

training, cybersecurity, and governance 

2: 

feedback from prior alerts, public awareness, and alert frequency 

3: 

availability, accessibility, reliability, and ease of use 

4: 

timeliness of dissemination, message understandability, and accuracy 

5: 

urgency, severity, and certainty 

6 

geographic breadth 

7: time of day 


For example, the first scenario addresses training, cybersecurity, and governance factors. The scenario 
presents four cases identifying different combinations of these factors 

• Case 1: Sufficient training, sufficient cybersecurity, insufficient governance 

• Case 2: Sufficient training, insufficient cybersecurity issues, sufficient governance 

• Case 3: Insufficient training, sufficient cybersecurity issues, sufficient governance 

• Case 4: Insufficient training, insufficient cybersecurity issues, insufficient governance 

Our analysis examines these permutations to extract the influence of each of the factors. 

For each of the cases of each of the scenarios, we ask that you indicate the likelihood of using the 
WEA service to issue public alerts. 


CMU/SEI-2013-SR-021 | 317 













TRUST MODEL VALIDATION SCENARIOS 


Scenario 1. This set of scenarios probes the impact of trainins, cvbersecuritv, and sovernance on 

your choice to use WEA. 

Your agency chose to deploy WEA to issue public alerts 16 months ago. The system is fully func¬ 
tional and, through training and practice you are fully capable of issuing alerts. 

Thunderstorms have been occumng across your city for the past 24 hours. NWS has responsibility 
for generating alerts for weather-related events, and has issued an alert for flash flooding of sever¬ 
al low-lying areas within the city. 

You receive reliable and confirmed information that a dam break on the Little Bear reservoir is 
imminent. This will result in major flooding downstream. 

For each case below, please indicate how likely you are to use WEA to issue a public alert under 
the stated circumstances? 


Case 1: sufficient training, sufficient cybersecurity, insufficient povernance _ 

You have been fully trained on the use of WEA, and understand both the principles behind 
WEA and the operation of the alerting software used in your agency. You have successfully 
practiced issuing alerts with your alerting software. You have templates for alerts that can be 
applied to this event. 

Over the past 12 months, no cvbersecuritv breaches of the WEA service (e.g., access by unau¬ 
thorized personnel, issuance of false alerts) have occurred in your agency, and none have been 
reported by other agencies 

You have an agreement with the local NWS office regarding the division of responsibilities for 
alerting. This agreement clearly defines that NWS will issue alerts for all weather related emer¬ 
gencies, and your agency will issue alerts for all other imminent threats. However, since the 
imminent dam break is the result of the on-going thunderstorms, you are unsure of your respon¬ 
sibility for issuing an alert for this threat. 


How likely are you to use WEA to issue a public alert under the circumstances stated above? 
(Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

(<5%) 

Not 

(5-20%) 

Not 

(20-40%) 

(40-60%) 

(60-80%) 

Probably 

(80-95%) 

(>95%) 
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Case 2: sufficient training, insufficient cybersecurity, sufficient governance _ 

You have been fully trained on the use of WEA, and understand both the principles behind 
WEA and the operation of the alerting software used in your agency. You have successfully 
practiced issuing alerts with your alerting software. You have templates for alerts that can be 
applied to this event. 

Over the past 12 months, several cvbersecurity breaches of the WEA service (e.g., access by 
unauthorized personnel, issuance of false alerts) have occurred in your agency, and others have 
been reported by other agencies. 

You have an agreement with the local NWS office regarding the division of responsibilities for 
alerting. This agreement clearly defines that NWS will issue alerts for all weather related emer¬ 
gencies, and your agency will issue alerts for all other imminent threats. Since the imminent 
dam break is not a weather event, you are sure of your responsibility for issuing an alert for this 
threat. 


How likely are you to use WEA to issue a public alert under the circumstances stated above? 
(Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

(<5%) 

Not 

(5-20%) 

Not 

(20-40%) 

(40-60%) 

(60-80%) 

Probably 

(80-95%) 

(>95%) 


Case 3: insufficient training, sufficient cybersecurity, sufficient governance _ 

You have not been fully trained on the use of WEA, and have only a basic understanding of 
both the principles behind WEA and the operation of the alerting software used in your agency. 
You have not had opportunities to practice issuing alerts with your alerting software. You do 
not have templates for alerts that can be applied to this event. 

Over the past 12 months, no cvbersecurity breaches of the WEA service (e.g., access by unau¬ 
thorized personnel, issuance of false alerts) have occurred in your agency, and none have been 
reported by other agencies 

You have an agreement with the local NWS office regarding the division of responsibilities for 
alerting. This agreement clearly defines that NWS will issue alerts for all weather related emer¬ 
gencies, and your agency will issue alerts for all other imminent threats. Since the imminent 
dam break is not a weather event, you are sure of your responsibility for issuing an alert for this 
threat. 


How likely are you to use WEA to issue a public alert under the circumstances stated above? 
(Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

(<5%) 

Not 

(5-20%) 

Not 

(20-40%) 

(40-60%) 

(60-80%) 

Probably 

(80-95%) 

(>95%) 
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Case 4: insufficient training, insufficient cybersecurity, insufficient governance _ 

You have not been fully trained on the use of WEA, and have only a basic understanding of 
both the principles behind WEA and the operation of the alerting software used in your agency. 
You have not had opportunities to practice issuing alerts with your alerting software. You do 
not have templates for alerts that can be applied to this event. 

Over the past 12 months, several cvbersecurity breaches of the WEA service (e.g., access by 
unauthorized personnel, issuance of false alerts) have occurred in your agency, and others have 
been reported by other agencies. 

You have an agreement with the local NWS office regarding the division of responsibilities for 
alerting. This agreement clearly defines that NWS will issue alerts for all weather related emer¬ 
gencies, and your agency will issue alerts for all other imminent threats. However, since the 
imminent dam break is the result of the on-going thunderstorms, you are unsure of your respon¬ 
sibility for issuing an alert for this threat. 


How likely are you to use WEA to issue a public alert under the circumstances stated above? 
(Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

(<5%) 

Not 

(5-20%) 

Not 

(20-40%) 

(40-60%) 

(60-80%) 

Probably 

(80-95%) 

(>95%) 
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Scenario 2. This set of scenarios probes the impact of feedback from prior alerts, public aware¬ 
ness, and alert frequency on your choice to use WEA. 

Your agency chose to deploy WEA to issue public alerts 6 months ago. The system is fully func¬ 
tional, and through training and practice you are fully capable of issuing alerts. 

For each case below, please indicate how likely you are to use WEA to issue a public alert for an 
imminent threat under the stated circumstances? 


Case 1: favorable feedback, public outreach, many prior alerts _ 

The public has responded to prior alerts with calls to 9-1-1 operators, and calls and emails to the 
EMA, the mayor, and the city council. 47% of the callers expressed questions about the WEA 
service (e.g., what is this service? What should I do? Why am I getting these alerts? How does 
WEA know where I am?), 42% expressed appreciation for the alerts received, and 11% object¬ 
ed to receiving the alerts. 

Prior to deploying WEA, your agency performed public outreach via TV, radio, and newspa¬ 
pers to inform the public about the upcoming alerting service. 

Within your jurisdiction over the past 6 months, the NWS has used WEA to issue 48 weather- 
related (e.g., tornadoes, thunderstorms) alerts. No AMBER alerts have been issued. Your agen¬ 
cy has issued 18 alerts for various imminent threats (e.g., police actions, chemical spills, struc¬ 
ture fires). 


How likely are you to use WEA to issue a public alert under the circumstances stated above? 
(Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

(<5%) 

Not 

(5-20%) 

Not 

(20-40%) 

(40-60%) 

(60-80%) 

Probably 

(80-95%) 

(>95%) 
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Case 2: favorable feedback, no public outreach, few prior alerts _ 

The public has responded to prior alerts with calls to 9-1-1 operators, and calls and emails to the 
EMA, the mayor, and the city council. 47% of the callers expressed questions about the WEA 
service (e.g., what is this service? What should I do? Why am I getting these alerts? How does 
WEA know where I am?), 42% expressed appreciation for the alerts received, and 11% object¬ 
ed to receiving the alerts. 

Your agency has not performed public outreach via TV, radio, and newspapers to inform the 
public about the WEA service. 

Within your jurisdiction over the past 6 months, the NWS has used WEA to issue 8 weather- 
related (e.g., tornadoes, snowstorms! alerts . No AMBER alerts have been issued. Your agency 
has issued 3 alerts for imminent threats - one for a police action (armed standoff), one for a 
chemical spill, and one for a fire requiring a 4 square block evacuation. 


How likely are you to use WEA to issue a public alert under the circumstances stated above? 
(Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

(<5%) 

Not 

(5-20%) 

Not 

(20-40%) 

(40-60%) 

(60-80%) 

Probably 

(80-95%) 

(>95%) 


Case 3: unfavorable feedback, public outreach, few prior alerts _ 

The public has responded to prior alerts with calls to 9-1-1 operators, and calls and emails to the 
EMA, the mayor, and the city council. 47% of the callers expressed questions about the WEA 
service (e.g., what is this service? What should I do? Why am I getting these alerts? How does 
WEA know where I am?), 11% expressed appreciation for the alerts received, and 42% object¬ 
ed to receiving the alerts . 

Prior to deploying WEA, your agency performed public outreach via TV, radio, and newspa¬ 
pers to inform the public about the WEA service. 

Within your jurisdiction over the past 6 months, the NWS has used WEA to issue 8 weather- 
related (e.g., tornadoes, snowstorms! alerts . No AMBER alerts have been issued. Your agency 
has issued 3 alerts for imminent threats - one for a police action (armed standoff), one for a 
chemical spill, and one for a fire requiring a 4 square block evacuation. 


How likely are you to use WEA to issue a public alert under the circumstances stated above? 
(Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

(<5%) 

Not 

(5-20%) 

Not 

(20-40%) 

(40-60%) 

(60-80%) 

Probably 

(80-95%) 

(>95%) 
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Case 4: unfavorable feedback, no public outreach, many prior alerts _ 

The public has responded to prior alerts with calls to 9-1-1 operators, and calls and emails to the 
EMA, the mayor, and the city council. 47% of the callers expressed questions about the WEA 
service (e.g., what is this service? What should I do? Why am I getting these alerts? How does 
WEA know where I am?), 11% expressed appreciation for the alerts received, and 42% object¬ 
ed to receiving the alerts . 

Your agency has not performed public outreach via TV, radio, and newspapers to inform the 
public about the WEA service. 

Within your jurisdiction over the past 6 months, the NWS has used WEA to issue 48 weather- 
related (e.g., tornadoes, thunderstorms! alerts . No AMBER alerts have been issued. Your agen¬ 
cy has issued 18 alerts for various imminent threats (e.g., police actions, chemical spills, struc¬ 
ture fires). 


How likely are you to use WEA to issue a public alert under the circumstances stated above? 
(Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

(<5%) 

Not 

(5-20%) 

Not 

(20-40%) 

(40-60%) 

(60-80%) 

Probably 

(80-95%) 

(>95%) 
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Scenario 3. This set of scenarios probes the impact of availability, accessibility, reliability, and 

ease of use on your choice to use WEA. 

Your agency chose to deploy WEA to issue public alerts 6 months ago. The system is fully functional, 
and through training and practice you are fully capable of issuing alerts. 

For each case below, please indicate how likely you are to use WEA to issue a public alert for an im¬ 
minent threat under the stated circumstances. 


Case 1: high availability, high accessibility, high reliability, easy to use _ 

Over the past 12 months, the WEA service has been down for maintenance approximately 1 
hour per year. 

WEA alerts can be generated and issued from your office or from your smart phone using a se¬ 
cure app that interfaces to your alert generation software. 

Your alerting software and the WEA service successfully transmits 99.9% of alei~ts that are cor¬ 
rectly configured and entered. 

The process of using your alerting software to create and send an alert to IPAWS-OPEN re¬ 
quires approximately 5 minutes of effort. 


How likely are you to use WEA to issue a public alert under the circumstances stated above? 
(Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

(<5%) 

Not 

(5-20%) 

Not 

(20-40%) 

(40-60%) 

(60-80%) 

Probably 

(80-95%) 

(>95%) 
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Case 2: high availability, high accessibility, low reliability, difficult to use _ 

Over the past 12 months, the WEA service has been down for maintenance approximately i 
hour per year. 

WEA alerts can be generated and issued from your office or from your smart phone using a se¬ 
cure app that interfaces to your alert generation software. 

Your alerting software and the WEA service successfully transmits 90% of alerts that are cor¬ 
rectly configured and entered. 

The process of using your alerting software to create and send an alert to IPAWS-OPEN re¬ 
quires approximately 40 minutes of effort . 


How likely are you to use WEA to issue a public alert under the circumstances stated above? 
(Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

(<5%) 

Not 

(5-20%) 

Not 

(20-40%) 

(40-60%) 

(60-80%) 

Probably 

(80-95%) 

(>95%) 


Case 3: high availability, low accessibility, high reliability, difficult to use _ 

Over the past 12 months, the WEA service has been down for maintenance approximately i 
hour per year. 

WEA alerts can be generated and issued only from a dedicated terminal within your agency’s 
central office. 

Your alerting software and the WEA service successfully transmits 99.9% of alerts that are cor¬ 
rectly configured and entered. 

The process of using your alerting software to create and send an alert to IPAWS-OPEN re¬ 
quires approximately 40 minutes of effort. 


How likely are you to use WEA to issue a public alert under the circumstances stated above? 
(Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

(<5%) 

Not 

(5-20%) 

Not 

(20-40%) 

(40-60%) 

(60-80%) 

Probably 

(80-95%) 

(>95%) 
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Case 4: high availability, low accessibility, low reliability, easy to me _ 

Over the past 12 months, the WEA service has been down for maintenance approximately i 
hour per year. 

WEA alerts can be generated and issued only from a dedicated terminal within your agency’s 
central office. 

Your alerting software and the WEA service successfully transmits 90% of alerts that are cor¬ 
rectly configured and entered. 

The process of using your alerting software to create and send an alert to IPAWS-OPEN re¬ 
quires approximately 5 minutes of effort. 


How likely are you to use WEA to issue a public alert under the circumstances stated above? 
(Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

(<5%) 

Not 

(5-20%) 

Not 

(20-40%) 

(40-60%) 

(60-80%) 

Probably 

(80-95%) 

(>95%) 


Case 5: low availability, high accessibility, high reliability, difficult to use _ 

Over the past 12 months, the WEA service has been down for maintenance approximately i 
hour per week. 

WEA alerts can be generated and issued from your office or from your smart phone using a se¬ 
cure app that interfaces to your alert generation software. 

Your alerting software and the WEA service successfully transmits 99.9% of alerts that are cor¬ 
rectly configured and entered. 

The process of using your alerting software to create and send an alert to IPAWS-OPEN re¬ 
quires approximately 40 minutes of effort. 


How likely are you to use WEA to issue a public alert under the circumstances stated above? 
(Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

(<5%) 

Not 

(5-20%) 

Not 

(20-40%) 

(40-60%) 

(60-80%) 

Probably 

(80-95%) 

(>95%) 
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Case 6: low availability, high accessibility, low reliability, easy to me _ 

Over the past 12 months, the WEA service has be down for maintenance approximately 1 hour 
per week 

WEA alerts can be generated and issued from your office or from your smart phone using a se¬ 
cure app that interfaces to your alert generation software. 

Your alerting software and the WEA service successfully transmits 90% of alerts that are cor¬ 
rectly configured and entered. 

The process of using your alerting software to create and send an alert to IPAWS-OPEN re¬ 
quires approximately 5 minutes of effort. 


How likely are you to use WEA to issue a public alert under the circumstances stated above? 
(Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

(<5%) 

Not 

(5-20%) 

Not 

(20-40%) 

(40-60%) 

(60-80%) 

Probably 

(80-95%) 

(>95%) 


Case 7; low availability, low accessibility, high reliability, easy to me _ 

Over the past 12 months, the WEA service has be down for maintenance approximately 1 hour 
per week. 

WEA alerts can be generated and issued only from a dedicated terminal within your agency’s 
central office 

Your alerting software and the WEA service successfully transmits 99.9% of alerts that are cor¬ 
rectly configured and entered. 

The process of using your alerting software to create and send an alert to IPAWS-OPEN re¬ 
quires approximately 5 minutes of effort. 


How likely are you to use WEA to issue a public alert under the circumstances stated above? 
(Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

(<5%) 

Not 

(5-20%) 

Not 

(20-40%) 

(40-60%) 

(60-80%) 

Probably 

(80-95%) 

(>95%) 
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Case 8: low availability, low accessibility, low reliability, difficult to use _ 

Over the past 12 months, the WEA service has be down for maintenance approximately 1 hour 
per week. 

WEA alerts can be generated and issued only from a dedicated terminal within your agency’s 
central office. 

Your alerting software and the WEA service successfully transmits 90% of alerts that are cor¬ 
rectly configured and entered. 

The process of using your alerting software to create and send an alert to IPAWS-OPEN re¬ 
quires approximately 40 minutes of effort. 


How likely are you to use WEA to issue a public alert under the circumstances stated above? 
(Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

(<5%) 

Not 

(5-20%) 

Not 

(20-40%) 

(40-60%) 

(60-80%) 

Probably 

(80-95%) 

(>95%) 
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Scenario 4. This set of scenarios probes the impact of timeliness of dissemination, message under- 
standabilitv, and accuracy on your choice to use WE A. 

Your agency chose to deploy WEA to issue public alerts 6 months ago. The system is fully functional, 
and through training and practice you are fully capable of issuing alerts. 

For each case below, please indicate how likely you are to use WEA to issue a public alert for an im¬ 
minent threat under the stated circumstances. 


Case 1: rapid dissemination, high understandability, low accuracy _ 

Prior messages have been distributed within 2 minutes of your submitting the WEA data input. 

You can craft any message of your choosing with a maximum size of 270 characters. 

Prior WEA messages sent to IPAWS-OPEN have been disseminated with significant errors in 
the message data (errors that affect the understandability or content of the message). 


How likely are you to use WEA to issue a public alert under the circumstances stated above? 
(Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

(<5%) 

Not 

(5-20%) 

Not 

(20-40%) 

(40-60%) 

(60-80%) 

Probably 

(80-95%) 

(>95%) 


Case 2: rapid dissemination, low understandability, high accuracy _ 

Prior messages have been distributed within 2 minutes of your submitting the WEA data input. 

You are able to issue only “standardized” messages developed by the WEA service based on 
your CAP (Common Alerting Protocol) inputs. In other words, you may not tailor the message. 

Prior WEA messages sent to IPAWS-OPEN have been disseminated with no errors in the mes¬ 
sage data. 


How likely are you to use WEA to issue a public alert under the circumstances stated above? 
(Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

(<5%) 

Not 

(5-20%) 

Not 

(20-40%) 

(40-60%) 

(60-80%) 

Probably 

(80-95%) 

(>95%) 
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Case 3: slow dissemination, high understandability, high accuracy _ 

Prior messages have been distributed within 10 to 30 minutes of your submitting the WEA data 
input. 

You can craft any message of your choosing with a maximum size of 270 characters. 

Prior WEA messages sent to IPAWS-OPEN have been disseminated with no errors in the mes¬ 
sage data. 


How likely are you to use WEA to issue a public alert under the circumstances stated above? 
(Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

(<5%) 

Not 

(5-20%) 

Not 

(20-40%) 

(40-60%) 

(60-80%) 

Probably 

(80-95%) 

(>95%) 


Case 4: slow dissemination, low understandability, low accuracy _ 

Prior messages have been distributed within 10 to 30 minutes of your submitting the WEA data 
input. 

You are able to issue only “standardized” messages developed by the WEA service based on 
your CAP (Common Alerting Protocol) inputs. In other words, you may not tailor the message. 

Prior WEA messages sent to IPAWS-OPEN have been disseminated with significant errors in 
the message data (errors that affect the understandability or content of the message). 


How likely are you to use WEA to issue a public alert under the circumstances stated above? 
(Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

(<5%) 

Not 

(5-20%) 

Not 

(20-40%) 

(40-60%) 

(60-80%) 

Probably 

(80-95%) 

(>95%) 
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Scenario 5. This set of scenarios probes the impact of urgency, severity, and certainty on your 
choice to use WEA. 

Your agency chose to deploy WEA to issue public alerts 6 months ago. The system is fully functional, 
and through training and practice you are fully capable of issuing alerts. 

For each case below, please indicate how likely you are to use WEA to issue a public alert for an im¬ 
minent threat under the stated circumstances. 


Case 1: high urgency, high severity, low certainty _ 

For an event that requires public action within 10 minutes . 

For an event that poses extraordinary threat to life. 

For a severe and urgent event that is 30% likely to occur. 


How likely are you to use WEA to issue a public alert under the circumstances stated above? 
(Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

(<5%) 

Not 

(5-20%) 

Not 

(20-40%) 

(40-60%) 

(60-80%) 

Probably 

(80-95%) 

(>95%) 


Case 2: high urgency, low severity, high certainty _ 

For an event that requires public action within 10 minutes . 

For an event that poses possible threat to property. 

For a severe and urgent event that is 90% likely to occur. 


How likely are you to use WEA to issue a public alert under the circumstances stated above? 
(Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

(<5%) 

Not 

(5-20%) 

Not 

(20-40%) 

(40-60%) 

(60-80%) 

Probably 

(80-95%) 

(>95%) 
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Case 3: low urgency, high seventy, high certainty _ 

For an event that requires public action within 2 hours . 

For an event that poses extraordinary threat to life. 

For a severe and urgent event that is 90% likely to occur. 


How likely are you to use WEA to issue a public alert under the circumstances stated above? 
(Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

(<5%) 

Not 

(5-20%) 

Not 

(20-40%) 

(40-60%) 

(60-80%) 

Probably 

(80-95%) 

(>95%) 


Case 4: low urgency, low severity, low certainty _ 

For an event that requires public action within 2 hours . 

For an event that poses possible threat to property . 

For a severe and urgent event that is 30% likely to occur. 


How likely are you to use WEA to issue a public alert under the circumstances stated above? 
(Please choose one) 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

(<5%) 

Not 

(5-20%) 

Not 

(20-40%) 

(40-60%) 

(60-80%) 

Probably 

(80-95%) 

(>95%) 
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Scenario 6. This set of scenarios probes the impact of the geographic breadth of an event on your 
choice to use WEA. 

Your agency chose to deploy WEA to issue public alerts 6 months ago. The system is fully functional, 
and through training and practice you are fully capable of issuing alerts. 

Do you think WEA would be an appropriate tool to issue a public alert for a severe and urgent event... 


... where 30% of the alert recipients are in the hazard zone and 70% are outside the zone? 
Please choose one: 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

(<5%) 

Not 

(5-20%) 

Not 

(20-40%) 

(40-60%) 

(60-80%) 

Probably 

(80-95%) 

(>95%) 


... where 90% of the alert recipients are in the hazard zone and 10% are outside the zone? 
Please choose one: 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

Definitely 

Very Probably 

Probably 

Maybe 

Probably 

Very 

Definitely 

Not 

(<5%) 

Not 

(5-20%) 

Not 

(20-40%) 

(40-60%) 

(60-80%) 

Probably 

(80-95%) 

(>95%) 
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Scenario 7. This set of scenarios probes the impact of the time of day of an event on your choice 
to use WEA. 

Your agency chose to deploy WEA to issue public alerts 6 months ago. The system is fully functional, 
and through training and practice you are fully capable of issuing alerts. 

Do you think WEA would be an appropriate tool to issue a public alert for a severe and urgent event... 


... that occurs at 10:30 AM ? 
Please choose one: 


□ 

□ 

□ 

Definitely 

Very Probably 

Probably 

Not 

Not 

Not 

(<5%) 

(5-20%) 

(20-40%) 

occurs at 2:30 AM ? 


Please choose one: 


□ 

□ 

□ 

Definitely 

Very Probably 

Probably 

Not 

Not 

Not 

(<5%) 

(5-20%) 

(20-40%) 


□ 

□ 

□ 

□ 

Maybe 

Probably 

Very 

Probably 

Definitely 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 


□ 

□ 

□ 

□ 

Maybe 

Probably 

Very 

Probably 

Definitely 

(40-60%) 

(60-80%) 

(80-95%) 

(>95%) 
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Appendix AA Alert Originator Trust Model Validation 
Graphical Results 


Scenario 1 



1 2 3 4 5 6 7 


Figure 108: Case 1: Sufficient Training, Sufficient Cybersecurity, and Insufficient Governance 



1 2 3 4 5 6 7 


Figure 109: Case 2: Sufficient Training, Insufficient Cybersecurity, and Sufficient Governance 



1 2 3 4 5 6 7 


Figure 110: Case 3: Insufficient Training, Sufficient Cybersecurity, and Sufficient Governance 



1 2 3 4 S 6 7 


Figure 111: Case 4: Insufficient Training, Insufficient Cybersecurity, and Insufficient Governance 
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Scenario 2 
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Scenario 3 
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Figure 120: Case 5: Low Availability, High Accessibility, High Reliability, and Difficult to Use 



1 2 3 4 S 6 7 


Figure 121: Case 6: Low Availability, High Accessibility, Low Reliability, and Easy to Use 



12 3 4 5 6 7 


Figure 122: Case 7: Low Availability, Low Accessibility, High Reliability, and Easy to Use 



Figure 123: Case 8: Low Availability, Low Accessibility, Low Reliability, and Difficult to Use 
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Scenario 4 



Figure 124: Case 1: Rapid Dissemination, High Understandabiiity, and Low Accuracy 



Figure 125: Case 2: Rapid Dissemination, Low Understandabiiity, and High Accuracy 
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Figure 126: Case 3: Siow Dissemination, High Understandabiiity, and High Accuracy 


3.5 



1 2 3 4 5 6 7 


Figure 127: Case 4: Slow Dissemination, Low Understandabiiity, and Low Accuracy 
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Scenario 5 



Figure 128: Case 1: High Urgency, High Severity, and Low Certainty 



1 2 3 4 S 6 7 


Figure 129: Case 2: High Urgency, Low Severity, and High Certainty 


8 



1 J 3 8 5 6 7 


Figure 130: Case 3: Low Urgency, High Severity, and High Certainty 



Figure 131: Case 4: Low Urgency, Low Severity, and Low Certainty 
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Scenario 6 



Figure 132: Case 1: Small Event and Large Alert 



1 2 3 4 S 6 7 


Figure 133: Case 2: Large Event and Large Alert 

Scenario 7 



Figure 135: Case 2: 2:30 a.m. 
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Appendix AB Alert Originator Trust Model Validation 
Statistical Analysis 


Table 80: Validation of AO BBN Model Against Results of AO Validation Scenario Interviews 


Scenario 

Validation 
Score from 
Validation 
Interviews 

BBN Pre¬ 
diction of 
Scenario 

Outcome 
of Inter¬ 
est 

BBN Utiii- 

zation 

Prediction 

BBN 

Availabie 

Prediction 

BBN Effec¬ 
tive Pre¬ 
diction 

BBN Appro¬ 
priate Pre¬ 
diction 

1 

98 

40 

Utilization 

40 

73 

69 

71 

2 

98 

37 

Utilization 

37 

69 

69 

72 

3 

70 

40 

Utilization 

40 

73 

69 

72 

4 

30 

36 

Utilization 

36 

68 

69 

71 

5 

98 

39 

Utilization 

39 

71 

69 

72 

6 

98 

39 

Utilization 

39 

71 

69 

72 

7 

70 

38 

Utilization 

38 

71 

68 

72 

8 

70 

38 

Utilization 

38 

71 

67 

72 

9 

98 

41 

Utilization 

41 

74 

69 

72 

10 

50 

39 

Utilization 

39 

71 

69 

72 

11 

50 

36 

Utilization 

36 

68 

69 

72 

12 

88 

39 

Utilization 

39 

72 

69 

72 

13 

50 

39 

Utilization 

39 

71 

69 

72 

14 

88 

41 

Utilization 

41 

74 

69 

72 

15 

88 

39 

Utilization 

39 

72 

69 

72 

16 

30 

36 

Utilization 

36 

68 

69 

72 

17 

50 

37 

Utilization 

37 

71 

66 

72 

18 

98 

39 

Utilization 

39 

71 

69 

72 

19 

70 

39 

Utilization 

39 

71 

69 

72 

20 

30 

35 

Utilization 

35 

71 

63 

72 

21 

88 

39 

Utilization 

39 

71 

69 

72 

22 

70 

40 

Utilization 

40 

71 

69 

73 

23 

98 

40 

Utilization 

40 

71 

69 

73 

24 

13 

34 

Utilization 

34 

71 

69 

65 
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Scatterplot of Validation vs BBN Prediction 



Figure 136: Analysis of All Predictions Across All Outcome Nodes in Table 80 

Regression Analysis: Validation vs. BBN 

The regression equation is Validation = -357 + 11.1 BBN. 


Table 81: Regression Coefficient Results for Validation vs. BBN 


Predictor 

Coef 

SE Coef 

t 

P 

Constant 

-356.85 

74.46 

-4.79 

0.000 

BBN 

11.147 

1.940 

5.75 

0.000 


S = 17.2814, R-Sq = 60.0%, R-Sq(adj) = 58.2% 


Table 82: Analysis of Variance for Validation vs. BBN 


Source 

df 

SS 

MS 

F 

P 

Regression 

1 

9857.7 

9857.7 

33.01 

0.000 

Residual error 

22 

6570.2 

298.6 



Total 

23 

16428.0 
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Residual Plots for Validation 

Normal Probability Plot Versus Fits 
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Figure 137: Residual Plots for Regression of Overall Validation as Function of BBN Predictions 
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Appendix AC AgenaRisk Tool Configuration Settings 


During the course of the probabilistic trust modeling and simulation using AgenaRisk, we identi¬ 
fied a number of configuration settings as sensitive to proper model operation and simulation. 
Here we provide settings, warnings, and guidance that we deem critical to reproducing and run¬ 
ning the WEA trust models within AgenaRisk. 



Figure 138: Version of AgenaRisk Used for This Project 



Figure 139: Simuiations Settings Used for the Pubiic and AO Trust Modeis 
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Figure 140: Model Graph Properties Used for the Public and AO Trust Modeis 

We initially encountered some problems related to the Java runtime executable (JRE) size of the 
AgenaRisk models during simulation. Often, the simulation would fail when the JRE exceeded 
1 GB. As a result, we used the following JRE settings associated with the 32-bit Windows instal¬ 
lation of AgenaRisk. We altered the runtime parameters via Control Panel > Programs > Java 
(32 bit) > Java > View, as shown in Figure 141, to accommodate larger JRE sizes. 



Figure 141: Altering Runtime Parameters to Flandie Large JRE Sizes 
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